Les Stuck

image.png

When I began working in video – after learning the basics of lighting, camera, and editing – I became fascinated by compositing and special effects. This was ten years ago, back when most of this sort of thing was done in 2D. At the time, one of the most powerful compositing and effects applications was Shake, so I studied it extensively. It was node-based, like Jitter, but non-real-time. A couple of years after Shake was discontinued, the 

 object appeared, offering real-time node-based programming of 2D video–all done on the GPU! It quickly became my favorite object.

It’s not really a single object; it’s a special kind of Gen patcher environment for processing openGL textures which includes about 130 Gen-only operators (An operator is the Gen equivalent to a Max/MSP/Jitter object) . Most operators (about 100) perform standard math functions, so you may already have an idea what some of them do. There are, however, about 30 which are unique to Gen in video, and thus worthy of special study. They fall into three broad categories: color, coordinates, and control. So learning 

My goal in this article is to present a pathway for you to learn 

. I’ll be skimming along, sketching concepts from beginner to advanced, and pointing you to useful but sometimes hard-to-find learning resources. Before we begin, there are two things to note.

Let’s get a feel for the different words that are used to describe roughly the same things. Video is a stream of still images, called frames, consisting of a rectangular arrangement of pixels. Each frame has a width in pixels, and a height in pixels. Typical dimensions are 1920x1080 and 4096x2160. Each pixel has 3 or 4 numbers which specify its color–a mix of red, green, blue, and sometimes alpha (transparency). The color values are typically called channels. Each pixel has color values and location coordinates.

The Jitter matrix can store a single frame of video, but it can do much more. It can contain openGL geometries, audio data, and who knows what else. For our purposes, however, a Jitter matrix is a frame of video; its cells are pixels, and its planes are color channels.

In addition, openGL has its own terminology. 2D images were originally used to create something resembling a texture on the surface of a virtual 3D object, so a 2D image or stream of images is called a texture. Pixels are often called cells of the texture. The list of numbers specifying a color is called a vector, and the individual numbers are called indices (plural of index) or elements of the vector. A single-element vector is called a scalar.

To make talking about all this a little less cumbersome, let’s gloss over some details and assume that each of the rows below consists of approximate synonyms:

Finally, keep in mind that, in OJ (original Jitter) the alpha channel comes first: ARGB. In openGL, and for the rest of this article, alpha is always last: RGBA.

An important thing to remember when working with Max’s openGL objects (a.k.a. OB3D objects) is this: once your images are sent to the graphics processor (the GPU), it’s a big performance loss to bring them back to the CPU. To avoid that, connect the outlets of 

 to other OB3D objects. The final result can be rendered to a 

, but it’s usually a mistake to connect connect a 

So beware, Maxer - I break this rule in many of the help files that accompany this tutorial.

 file of example patches found at the bottom of this tutorial, uncompress, and look inside at the patches. They will be used often in this article. Anyway, to get a feel for how costly this technique is, take a look at the

patch. Note how bad the frame rate is. Then, delete the connections to those

 objects and see how the frame rate jumps up. Clearly, there’s a big unnecessary cost - the only time you should use this technique is for debugging (the same way you would use the “

” object) or for learning, which is what we are about to do.

 is a Jitter matrix or an openGL texture, the inlet makes sure it’s a texture inside, and that the Gen patch processes each pixel of that texture. Here, it’s a single pixel of a texture that arrives at “in 1” and is sent out “out 1”. The same image arrives at “in 2” where its pixels are multiplied by “param brightness” and sent out “out 2”. The 

 followed by a float or list) sent to any inlet of 

 object, splits a list into individual values, a vector into its indices, or a pixel value into its individual colors (all roughly equivalent terminology). The indices of the vector are numbered starting at 0, so “

” gives us the red value. Conveniently, you can also use “

” operator, it will not have one outlet for each color. Instead, you get a 3-element vector. You can use 

” uses the red value to replace the red, green, and blue values, resulting in a grayscale image. This is similar to typing “

 box. If you actually want to split the individual colors, you’ll need a separate 

 operator can assemble scalars into vectors. (The 

 operator, vec’s cousin, can assemble two vectors into a third.) Like 

 can also store an initial vector as its arguments. You would have to send a 

 to make it output the default list; in the Gen world that’s neither necessary nor possible. There are no 

 messages - or any other Max messages – in Gen. The vector is output each time a pixel is calculated, whether its value changes or not.

 patch, we can get its location (x,y) with the 

 operator, which outputs a two-index vector of horizontal and vertical pixels. However, it’s usually more convenient to use the 

 operator. This outputs a vector consisting of the horizontal coordinate, ranging from 0 to 1 (left to right) and the vertical coordinate, ranging from 0 to 1 (top to bottom) regardless of the pixel dimensions of the texture. The

 operator, it is able to access the entire texture. The right inlet to 

 receives a two-element position vector that is connected to a 

 or other operators calculating position coordinates. Then, the 

 operator’s output is the pixel vector at that other position. It’s your go-to repositioning tool.

At this point, you have an overview of how these operators work, but your head may be spinning. Fear not. Now it's time to take a leisurely stroll through the eloquent gardens of Gregory Taylor’s 

 operator. Working through these two articles might take a while, but it’s time spent spent. When done, you will have an understanding of the fine art of psychedelic colors and kaleidoscopic imagery. A career as VJ is not far behind.

You've earned a little fun. Check out this 

 made by Federico Foderaro. Watch from the beginning up to about 4:40. When he starts putting objects inside 

, around 3:30, you should be able to follow along and really understand what he’s doing. That warm glow you feel is pure pixel empowerment.

To get started on the intermediate path, open the

 Help > Examples > jitter- examples > gen > jit.gl.pix.sprinkle.maxpat

. patch and try to figure this patch out on your own. My 

 operator enables us to move discrete pixels around with the video noise. The 

 operator does vector math much like the Max 

 object. It has a little mathematical expression as its argument, which takes the input value, multiplies it by 2, then subtracts 1. The 

, it can also serve as an argument to the 

The other interesting thing about this patch is that you can’t unlock it and edit it the usual way. When 

 attribute, it loads a file with that name, in this case “sprinkle.genjit”, which is located in the Max search path (in this case, the same folder as the patch itself). This a great way to save Gen patchers for future use. After building an effect you’d like to use again, open the 

to create your very own genjit. To learn more about how you can use readymade 

 abstractions, and how their params appear the 

 and search for “Video and Graphics Tutorial 9”.

Now that you’re an intermediate user, you’re going to have to figure things out on your own more and more. Here are some resources to help with that. First of all, the 

. Surprisingly, this is not that useful for us right now. It’s mostly messages and attributes for the 

 object itself; we are more concerned with the Gen patching inside.

 explains all those math functions that you can use throughout the Gen universe. Take a few moments now and look at each one, maybe try out a few that intrigue you. Federico Foderaro puts the math functions to good use in this 

 Aren’t you glad you didn’t sleep through trigonometry class?

. Here you should recognize a few operators we’ve looked at so far. These are unique to video processing. Note that, in 

, “nearest” is the same as “sample”, and “samplepix” and “nearestpix” don’t work at all. Read through the others carefully, and try at least three on your own.

Whoa, way too much work! Let’s play with some patches. Open 

Help > Examples > jitter- examples > gen > 

 and play with the fourth element (alpha) of erase_color until you get a smearing effect. Note how an empty 

 object is used to make a feedback loop around a 

Good things come in Packages. Use Package Manager to download and install Andrew Benson’s fabulous 

, one of my all-time favorites. Andrew is the master of video feedback, and you can see how he uses 

 to render several objects, capture that into a texture, and feed it back into a 

 make a copy of the incoming textures, which prevents feedback from becoming an infinite loop.

Use the Package Manager again to install 2K’s awesome 

. Launch it and, in the examples tab, click on wfg, and control-click to see a new view of 

. You can also just type “vs_wfg_s” into an object box and double-click. Take it out of Presentation mode, and you’ll see “

” It’s a video oscillator! Now you can channel your inner Bill Etra and become a VJ for a Kraftwerk cover band.

Congratulations, now you’re an intermediate pixel pusher. We’ve covered a lot of material, but there’s one more thing...

Now that you are entering Level 3, it’s time to embrace the semicolon. Open 

Oh no, it’s a box of code!! Didn’t we start this whole thing saying how node-based video effects processing was the way to go? Yes, for passing images and pixels around, it’s perfect, but if you want to do elaborate math, you’ll be happier in the long run if you use some text. Sometimes, all you need is a little algorithm to add functionality that would be difficult to achieve with boxes and patch cords. And there’s no simpler way to code media software than Max’s Gen family. Just look at that single line “out1 = in1 + in2” - it’s even easier than JavaScript.

To get started on solid ground with all this, now is the time to read through several selections of the 

. Start with “Gen Operators”, and read through “Subpatchers and Abstractions”, stopping before “The 

 Object”. (Ignore the “history” operator - that makes feedback possible for the audio-only 

 operator.) This is an excellent tutorial, so make sure you work through all of it. Continue your reading from “Jitter Operators” through to the end of the article. If your brain is a little tired, that’s just what it feels like to be advanced. Be sure to get enough sleep.

Help > Examples > jitter-examples > gen >

 demonstrating function definitions, and well as a code version of the 

 operator, including an attribute. For a glimpse of just how spectacular code can be, open 

. Even though jit.gl.pix is outputting a 2D image, the code inside makes the image look 3D.

A note about nomenclature: If you’ve already done some work with the 

 objects, the conventions for inlet names might be a bit confusing. Open the “inlets_and_spaces” tab of 

 patch to see everything all laid out. While you're there, look through each of the other tabs to play with 

’s functionality and modify it. It’s worth noting that using swizzle notation inside the 

”). Finally, note that “Param” must be upper-case inside 

 is your final homework assignment. Put on your thinking cap, study each section carefully, then make a couple of simple patches with codebox.

And...you’re done. You may notice that it’s taken a couple of weeks to finish this article. (It might be time to do some laundry.) But now you’re an expert in building openGL fragment shaders with Max patches. The world is yours. Power to the pixel!

Check out the entire "My Favorite Object" series: 

pixhelp.zip

When I began working in video – after learning the basics of lighting, camera, and editing – I became fascinated by compositing and special effects. This was ten years ago, back when most of this sort of thing was done in 2D. At the time, one of the most powerful compositing and effects applications was Shake, so I studied it extensively. It was node-based, like Jitter, but non-real-time. A couple of years after Shake was discontinued, the jit.gl.pix object appeared, offering real-time node-based programming of 2D video–all done on the GPU! It quickly became my favorite object.

It’s not really a single object; it’s a special kind of Gen patcher environment for processing openGL textures which includes about 130 Gen-only operators (An operator is the Gen equivalent to a Max/MSP/Jitter object) . Most operators (about 100) perform standard math functions, so you may already have an idea what some of them do. There are, however, about 30 which are unique to Gen in video, and thus worthy of special study. They fall into three broad categories: color, coordinates, and control. So learning jit.gl.pix is not as difficult as it might seem.

My goal in this article is to present a pathway for you to learn jit.gl.pix. I’ll be skimming along, sketching concepts from beginner to advanced, and pointing you to useful but sometimes hard-to-find learning resources. Before we begin, there are two things to note.

An important thing to remember when working with Max’s openGL objects (a.k.a. OB3D objects) is this: once your images are sent to the graphics processor (the GPU), it’s a big performance loss to bring them back to the CPU. To avoid that, connect the outlets of jit.gl.pix only to other OB3D objects. The final result can be rendered to a jit.window or jit.pwindow object using either jit.world or jit.gl.render, but it’s usually a mistake to connect connect a jit.gl.pix directly to a jit.window, jit.pwindow, or a jit.matrix object.

So beware, Maxer - I break this rule in many of the help files that accompany this tutorial. 

Download the .zip file of example patches found at the bottom of this tutorial, uncompress, and look inside at the patches. They will be used often in this article. Anyway, to get a feel for how costly this technique is, take a look at the jit.gl.pix.norm+snorm+cell.maxhelp patch. Note how bad the frame rate is. Then, delete the connections to those jit.matrix objects and see how the frame rate jumps up. Clearly, there’s a big unnecessary cost - the only time you should use this technique is for debugging (the same way you would use the “print” object) or for learning, which is what we are about to do.

Open the jit.gl.pix.in+out.maxhelp file, and double-click jit.gl.pix to look inside. Whether the input to jit.gl.pix is a Jitter matrix or an openGL texture, the inlet makes sure it’s a texture inside, and that the Gen patch processes each pixel of that texture. Here, it’s a single pixel of a texture that arrives at “in 1” and is sent out “out 1”. The same image arrives at “in 2” where its pixels are multiplied by “param brightness” and sent out “out 2”. The param operator accepts Max messages (param name followed by a float or list) sent to any inlet of jit.gl.pix.

Now look at jit.gl.pix.swiz.maxhelp. The swiz operator, much like Max’s unpack object, splits a list into individual values, a vector into its indices, or a pixel value into its individual colors (all roughly equivalent terminology). The indices of the vector are numbered starting at 0, so “swiz 0” gives us the red value. Conveniently, you can also use “swiz r”. Unlike unpack, if you create a “swiz r g b” operator, it will not have one outlet for each color. Instead, you get a 3-element vector. You can use swiz to rearrange the color channels; “swiz r r r” uses the red value to replace the red, green, and blue values, resulting in a grayscale image. This is similar to typing “$1 $1 $1” into a message box. If you actually want to split the individual colors, you’ll need a separate swiz operator for each color.

Check out jit.gl.pix.vec.maxhelp file. Just as swiz can unpack a vector into scalars, the vec operator can assemble scalars into vectors. (The concat operator, vec’s cousin, can assemble two vectors into a third.) Like pack, vec can also store an initial vector as its arguments. You would have to send a bang message to pack to make it output the default list; in the Gen world that’s neither necessary nor possible. There are no bang messages - or any other Max messages – in Gen. The vector is output each time a pixel is calculated, whether its value changes or not. 

For each pixel that is processed by our jit.gl.pix patch, we can get its location (x,y) with the cell operator, which outputs a two-index vector of horizontal and vertical pixels. However, it’s usually more convenient to use the norm operator. This outputs a vector consisting of the horizontal coordinate, ranging from 0 to 1 (left to right) and the vertical coordinate, ranging from 0 to 1 (top to bottom) regardless of the pixel dimensions of the texture. The jit.gl.pix.norm+snorm+cell.maxhelp patch is your friend here.

Finally, check out the jit.gl.pix.sample+nearest.maxhelp patch. When an inlet is connected to the left inlet of a sample operator, it is able to access the entire texture. The right inlet to sample receives a two-element position vector that is connected to a norm or other operators calculating position coordinates. Then, the sample operator’s output is the pixel vector at that other position. It’s your go-to repositioning tool.

At this point, you have an overview of how these operators work, but your head may be spinning. Fear not. Now it's time to take a leisurely stroll through the eloquent gardens of Gregory Taylor’s The Joy of Swiz and Adventures in Vectorland. (Note that Gen now includes the selector operator.) Check out the jit.gl.pix.mix+selector+?.maxhelp patch to better understand the ? operator. Working through these two articles might take a while, but it’s time spent spent. When done, you will have an understanding of the fine art of psychedelic colors and kaleidoscopic imagery. A career as VJ is not far behind.

You've earned a little fun. Check out this real-time patching video made by Federico Foderaro. Watch from the beginning up to about 4:40. When he starts putting objects inside jit.gl.pix, around 3:30, you should be able to follow along and really understand what he’s doing. That warm glow you feel is pure pixel empowerment.

To get started on the intermediate path, open the Help > Examples > jitter- examples > gen > jit.gl.pix.sprinkle.maxpat. patch and try to figure this patch out on your own. My jit.gl.pix.mix+selector+?.maxhelp patch will explain the mix operator. Using the cell operator enables us to move discrete pixels around with the video noise. The expr operator does vector math much like the Max vexpr object. It has a little mathematical expression as its argument, which takes the input value, multiplies it by 2, then subtracts 1. The param operator not only receives param messages to jit.gl.pix, it can also serve as an argument to the < operator.

The other interesting thing about this patch is that you can’t unlock it and edit it the usual way. When jit.gl.pix has the @gen attribute, it loads a file with that name, in this case “sprinkle.genjit”, which is located in the Max search path (in this case, the same folder as the patch itself). This a great way to save Gen patchers for future use. After building an effect you’d like to use again, open the jit.gl.pix window and select File > Save As... to create your very own genjit. To learn more about how you can use readymade jit.gl.pix abstractions, and how their params appear the attrui object’s menu, select the menu item Help > Reference and search for “Video and Graphics Tutorial 9”.

Now that you’re an intermediate user, you’re going to have to figure things out on your own more and more. Here are some resources to help with that. First of all, the jit.gl.pix documentation page. Surprisingly, this is not that useful for us right now. It’s mostly messages and attributes for the jit.gl.pix object itself; we are more concerned with the Gen patching inside.

On the other hand, the Common Operators page explains all those math functions that you can use throughout the Gen universe. Take a few moments now and look at each one, maybe try out a few that intrigue you. Federico Foderaro puts the math functions to good use in this procedural drawing tutorial Aren’t you glad you didn’t sleep through trigonometry class?

The next really useful reference is the Gen Jitter Operators page. Here you should recognize a few operators we’ve looked at so far. These are unique to video processing. Note that, in jit.gl.pix, “nearest” is the same as “sample”, and “samplepix” and “nearestpix” don’t work at all. Read through the others carefully, and try at least three on your own.

Whoa, way too much work! Let’s play with some patches. Open Help > Examples > jitter- examples > gen > smear.o.vision.maxpat and play with the fourth element (alpha) of erase_color until you get a smearing effect. Note how an empty jit.gl.slab object is used to make a feedback loop around a jit.gl.pix.

Good things come in Packages. Use Package Manager to download and install Andrew Benson’s fabulous Jitter Recipes, then open SceneWarp, one of my all-time favorites. Andrew is the master of video feedback, and you can see how he uses jit.gl.node to render several objects, capture that into a texture, and feed it back into a jit.gl.pix. Both jit.gl.slab and jit.gl.pix make a copy of the incoming textures, which prevents feedback from becoming an infinite loop.

Use the Package Manager again to install 2K’s awesome Vsynth. Launch it and, in the examples tab, click on wfg, and control-click to see a new view of wfg. You can also just type “vs_wfg_s” into an object box and double-click. Take it out of Presentation mode, and you’ll see “jit.gl.pix vsynth @title oscillator” It’s a video oscillator! Now you can channel your inner Bill Etra and become a VJ for a Kraftwerk cover band.

Now that you are entering Level 3, it’s time to embrace the semicolon. Open jit.gl.pix.expr+codebox.maxhelp and look inside the “basic” jit.gl.pix object.

To get started on solid ground with all this, now is the time to read through several selections of the Gen Overview. Start with “Gen Operators”, and read through “Subpatchers and Abstractions”, stopping before “The gen~ Object”. (Ignore the “history” operator - that makes feedback possible for the audio-only gen~ operator.) This is an excellent tutorial, so make sure you work through all of it. Continue your reading from “Jitter Operators” through to the end of the article. If your brain is a little tired, that’s just what it feels like to be advanced. Be sure to get enough sleep.

Help > Examples > jitter-examples > gen > jit.gl.pix.pinch.maxpat includes a simple codebox demonstrating function definitions, and well as a code version of the sample operator, including an attribute. For a glimpse of just how spectacular code can be, open Help > Examples > jitter-examples > gen > julia.quat.raytracer.maxpat. Even though jit.gl.pix is outputting a 2D image, the code inside makes the image look 3D.

A note about nomenclature: If you’ve already done some work with the vexpr or jit.expr objects, the conventions for inlet names might be a bit confusing. Open the “inlets_and_spaces” tab of jit.gl.pix.expr+codebox.maxhelp patch to see everything all laid out. While you're there, look through each of the other tabs to play with codebox’s functionality and modify it. It’s worth noting that using swizzle notation inside the expr and codebox operators works just like the swiz operator (i.e., “in1.r” instead of “swiz r”). Finally, note that “Param” must be upper-case inside codebox.

For a complete understanding of how the expr and codebox operators work, the GenExpr page is your final homework assignment. Put on your thinking cap, study each section carefully, then make a couple of simple patches with codebox.

Check out the entire "My Favorite Object" series: bucket, stutter~, jstrigger, multislider, plot~, trigger, rate~, jit.slide, curve~, jit.gl.mesh, button, jit.gl.pix.

Learn More: See all the articles in this series

myfavoriteobject-jitglpix.jpg

The jit.gl.pix object has become a go-to object for my visual work. I’d like to present a pathway for you to develop your own skills with the object and with the Gen environment - skimming along, sketching concepts (from beginner to advanced), and pointing you to useful but sometimes hard-to-find learning resources.

my-favorite-object-jit-gl-pix

 how it works. Okay. See you guys in a week or so, then....

<facepalm> Oh. So that's how it works. Okay. See you guys in a week or so, then....

Thanks a lot for this great introduction! Especially your helpfile - patches are a rich source for demonstrating how to handle visual input in the gen - world.

I've been away from Max/MSP/Jitter for almost 2 years 😥 ...the audio stuff comes back like it was yesterday, but the visual was a struggle... until i found this article again, it really links everything together all in one place too(what with links to other sources of info and patches.... proves boundless in scope of how much information stems from this one hub), one of the best reads on this site. 

I've been away from Max/MSP/Jitter for almost 2 years 😥 ...the audio stuff comes back like it was yesterday, but the visual was a struggle... until i found this article again, it really links everything together all in one place too(what with links to other sources of info and patches.... proves boundless in scope of how much information stems from this one hub), one of the best reads on this site. 
thank you so much!

I found this to be very helpful and concise. not sure if it's the upgrade or not but one thing was different from what you described with the jit.gl.pix.norm+snorm+cell. I got ~60 fps when I first opened patch and it stayed around there after removing all the wires to matrices.

 is not included in pixhelp.zip . is it available?

also, jit.gl.pix.mix+selector+?.maxhelp is not included in pixhelp.zip . is it available?

i have jit.gl.pix.mix+selector+?.maxhelp in my zip file, though no ***ji.gl.pix.sprinkle :/***

Candace, shoot me a message if you need i'll send you on the mix ? thing.

If you just go into gl.pix and create mix and ? objects, theyre actually really self explanatory :)

*** its on my computer, it wasnt showing up in the max help search either, but i just spotlighted it and found it. absolute newb error :P

this is so awesome!!!
i have  jit.gl.pix.mix+selector+?.maxhelp in my zip file, though no ***ji.gl.pix.sprinkle :/***
Candace, shoot me a message if you need i'll send you on the mix ? thing.

If you just go into gl.pix and create mix and ? objects, theyre actually really self explanatory :)

*** its on my computer, it wasnt showing up in the max help search either, but i just spotlighted it and found it. absolute newb error :P


Ye, I understand better jit.gl.pix, now! I have a question! Do you know if there is a way to control z axe of a mesh with luma param in jit.gl.pix? Like the gen in the help of jit.gl.mesh... Thanks

Ye, I understand better jit.gl.pix, now!  I have a question! Do you know if there is a way to control z axe of a mesh with luma param in jit.gl.pix? Like the gen in the help of jit.gl.mesh... Thanks

My Favorite Object: jit.gl.pix