My Favorite Object: jit.gl.pix

When I began working in video – after learning the basics of lighting, camera, and editing – I became fascinated by compositing and special effects. This was ten years ago, back when most of this sort of thing was done in 2D. At the time, one of the most powerful compositing and effects applications was Shake, so I studied it extensively. It was node-based, like Jitter, but non-real-time. A couple of years after Shake was discontinued, the jit.gl.pix object appeared, offering real-time node-based programming of 2D video–all done on the GPU! It quickly became my favorite object.

It’s not really a single object; it’s a special kind of Gen patcher environment for processing openGL textures which includes about 130 Gen-only operators (An operator is the Gen equivalent to a Max/MSP/Jitter object) . Most operators (about 100) perform standard math functions, so you may already have an idea what some of them do. There are, however, about 30 which are unique to Gen in video, and thus worthy of special study. They fall into three broad categories: color, coordinates, and control. So learning jit.gl.pix is not as difficult as it might seem.

My goal in this article is to present a pathway for you to learn jit.gl.pix. I’ll be skimming along, sketching concepts from beginner to advanced, and pointing you to useful but sometimes hard-to-find learning resources. Before we begin, there are two things to note.

Fuzzy Terminology

Let’s get a feel for the different words that are used to describe roughly the same things. Video is a stream of still images, called frames, consisting of a rectangular arrangement of pixels. Each frame has a width in pixels, and a height in pixels. Typical dimensions are 1920x1080 and 4096x2160. Each pixel has 3 or 4 numbers which specify its color–a mix of red, green, blue, and sometimes alpha (transparency). The color values are typically called channels. Each pixel has color values and location coordinates.

The Jitter matrix can store a single frame of video, but it can do much more. It can contain openGL geometries, audio data, and who knows what else. For our purposes, however, a Jitter matrix is a frame of video; its cells are pixels, and its planes are color channels.

In addition, openGL has its own terminology. 2D images were originally used to create something resembling a texture on the surface of a virtual 3D object, so a 2D image or stream of images is called a texture. Pixels are often called cells of the texture. The list of numbers specifying a color is called a vector, and the individual numbers are called indices (plural of index) or elements of the vector. A single-element vector is called a scalar.

To make talking about all this a little less cumbersome, let’s gloss over some details and assume that each of the rows below consists of approximate synonyms:

Video : Jitter : openGL

frame = matrix = texture

pixel = cell = cell

channel = plane = index

color = cell value = vector

Finally, keep in mind that, in OJ (original Jitter) the alpha channel comes first: ARGB. In openGL, and for the rest of this article, alpha is always last: RGBA.

Caveat Maxtrix

An important thing to remember when working with Max’s openGL objects (a.k.a. OB3D objects) is this: once your images are sent to the graphics processor (the GPU), it’s a big performance loss to bring them back to the CPU. To avoid that, connect the outlets of jit.gl.pix only to other OB3D objects. The final result can be rendered to a jit.window or jit.pwindow object using either jit.world or jit.gl.render, but it’s usually a mistake to connect connect a jit.gl.pix directly to a jit.window, jit.pwindow, or a jit.matrix object.

So beware, Maxer - I break this rule in many of the help files that accompany this tutorial.

Download the .zip file of example patches found at the bottom of this tutorial, uncompress, and look inside at the patches. They will be used often in this article. Anyway, to get a feel for how costly this technique is, take a look at the jit.gl.pix.norm+snorm+cell.maxhelp patch. Note how bad the frame rate is. Then, delete the connections to those jit.matrix objects and see how the frame rate jumps up. Clearly, there’s a big unnecessary cost - the only time you should use this technique is for debugging (the same way you would use the “print” object) or for learning, which is what we are about to do.

The Beginning Path

Open the jit.gl.pix.in+out.maxhelp file, and double-click jit.gl.pix to look inside. Whether the input to jit.gl.pix is a Jitter matrix or an openGL texture, the inlet makes sure it’s a texture inside, and that the Gen patch processes each pixel of that texture. Here, it’s a single pixel of a texture that arrives at “in 1” and is sent out “out 1”. The same image arrives at “in 2” where its pixels are multiplied by “param brightness” and sent out “out 2”. The param operator accepts Max messages (param name followed by a float or list) sent to any inlet of jit.gl.pix.

Now look at jit.gl.pix.swiz.maxhelp. The swiz operator, much like Max’s unpack object, splits a list into individual values, a vector into its indices, or a pixel value into its individual colors (all roughly equivalent terminology). The indices of the vector are numbered starting at 0, so “swiz 0” gives us the red value. Conveniently, you can also use “swiz r”. Unlike unpack, if you create a “swiz r g b” operator, it will not have one outlet for each color. Instead, you get a 3-element vector. You can use swiz to rearrange the color channels; “swiz r r r” uses the red value to replace the red, green, and blue values, resulting in a grayscale image. This is similar to typing “$1 $1 $1” into a message box. If you actually want to split the individual colors, you’ll need a separate swiz operator for each color.

Check out jit.gl.pix.vec.maxhelp file. Just as swiz can unpack a vector into scalars, the vec operator can assemble scalars into vectors. (The concat operator, vec’s cousin, can assemble two vectors into a third.) Like pack, vec can also store an initial vector as its arguments. You would have to send a bang message to pack to make it output the default list; in the Gen world that’s neither necessary nor possible. There are no bang messages - or any other Max messages – in Gen. The vector is output each time a pixel is calculated, whether its value changes or not.

For each pixel that is processed by our jit.gl.pix patch, we can get its location (x,y) with the cell operator, which outputs a two-index vector of horizontal and vertical pixels. However, it’s usually more convenient to use the norm operator. This outputs a vector consisting of the horizontal coordinate, ranging from 0 to 1 (left to right) and the vertical coordinate, ranging from 0 to 1 (top to bottom) regardless of the pixel dimensions of the texture. The jit.gl.pix.norm+snorm+cell.maxhelp patch is your friend here.

Finally, check out the jit.gl.pix.sample+nearest.maxhelp patch. When an inlet is connected to the left inlet of a sample operator, it is able to access the entire texture. The right inlet to sample receives a two-element position vector that is connected to a norm or other operators calculating position coordinates. Then, the sample operator’s output is the pixel vector at that other position. It’s your go-to repositioning tool.

At this point, you have an overview of how these operators work, but your head may be spinning. Fear not. Now it's time to take a leisurely stroll through the eloquent gardens of Gregory Taylor’s The Joy of Swiz and Adventures in Vectorland. (Note that Gen now includes the selector operator.) Check out the jit.gl.pix.mix+selector+?.maxhelp patch to better understand the ? operator. Working through these two articles might take a while, but it’s time spent spent. When done, you will have an understanding of the fine art of psychedelic colors and kaleidoscopic imagery. A career as VJ is not far behind.

You've earned a little fun. Check out this real-time patching video made by Federico Foderaro. Watch from the beginning up to about 4:40. When he starts putting objects inside jit.gl.pix, around 3:30, you should be able to follow along and really understand what he’s doing. That warm glow you feel is pure pixel empowerment.

The Intermediate Path

To get started on the intermediate path, open the Help > Examples > jitter- examples > gen > jit.gl.pix.sprinkle.maxpat. patch and try to figure this patch out on your own. My jit.gl.pix.mix+selector+?.maxhelp patch will explain the mix operator. Using the cell operator enables us to move discrete pixels around with the video noise. The expr operator does vector math much like the Max vexpr object. It has a little mathematical expression as its argument, which takes the input value, multiplies it by 2, then subtracts 1. The param operator not only receives param messages to jit.gl.pix, it can also serve as an argument to the < operator.

The other interesting thing about this patch is that you can’t unlock it and edit it the usual way. When jit.gl.pix has the @gen attribute, it loads a file with that name, in this case “sprinkle.genjit”, which is located in the Max search path (in this case, the same folder as the patch itself). This a great way to save Gen patchers for future use. After building an effect you’d like to use again, open the jit.gl.pix window and select File > Save As... to create your very own genjit. To learn more about how you can use readymade jit.gl.pix abstractions, and how their params appear the attrui object’s menu, select the menu item Help > Reference and search for “Video and Graphics Tutorial 9”.

Now that you’re an intermediate user, you’re going to have to figure things out on your own more and more. Here are some resources to help with that. First of all, the jit.gl.pix documentation page. Surprisingly, this is not that useful for us right now. It’s mostly messages and attributes for the jit.gl.pix object itself; we are more concerned with the Gen patching inside.

On the other hand, the Common Operators page explains all those math functions that you can use throughout the Gen universe. Take a few moments now and look at each one, maybe try out a few that intrigue you. Federico Foderaro puts the math functions to good use in this procedural drawing tutorial Aren’t you glad you didn’t sleep through trigonometry class?

The next really useful reference is the Gen Jitter Operators page. Here you should recognize a few operators we’ve looked at so far. These are unique to video processing. Note that, in jit.gl.pix, “nearest” is the same as “sample”, and “samplepix” and “nearestpix” don’t work at all. Read through the others carefully, and try at least three on your own.

Whoa, way too much work! Let’s play with some patches. Open Help > Examples > jitter- examples > gen > smear.o.vision.maxpat and play with the fourth element (alpha) of erase_color until you get a smearing effect. Note how an empty jit.gl.slab object is used to make a feedback loop around a jit.gl.pix.

Good things come in Packages. Use Package Manager to download and install Andrew Benson’s fabulous Jitter Recipes, then open SceneWarp, one of my all-time favorites. Andrew is the master of video feedback, and you can see how he uses jit.gl.node to render several objects, capture that into a texture, and feed it back into a jit.gl.pix. Both jit.gl.slab and jit.gl.pix make a copy of the incoming textures, which prevents feedback from becoming an infinite loop.

Use the Package Manager again to install 2K’s awesome Vsynth. Launch it and, in the examples tab, click on wfg, and control-click to see a new view of wfg. You can also just type “vs_wfg_s” into an object box and double-click. Take it out of Presentation mode, and you’ll see “jit.gl.pix vsynth @title oscillator” It’s a video oscillator! Now you can channel your inner Bill Etra and become a VJ for a Kraftwerk cover band.

Congratulations, now you’re an intermediate pixel pusher. We’ve covered a lot of material, but there’s one more thing...

The Advanced Path

Now that you are entering Level 3, it’s time to embrace the semicolon. Open jit.gl.pix.expr+codebox.maxhelp and look inside the “basic” jit.gl.pix object.

Oh no, it’s a box of code!! Didn’t we start this whole thing saying how node-based video effects processing was the way to go? Yes, for passing images and pixels around, it’s perfect, but if you want to do elaborate math, you’ll be happier in the long run if you use some text. Sometimes, all you need is a little algorithm to add functionality that would be difficult to achieve with boxes and patch cords. And there’s no simpler way to code media software than Max’s Gen family. Just look at that single line “out1 = in1 + in2” - it’s even easier than JavaScript.

To get started on solid ground with all this, now is the time to read through several selections of the Gen Overview. Start with “Gen Operators”, and read through “Subpatchers and Abstractions”, stopping before “The gen~ Object”. (Ignore the “history” operator - that makes feedback possible for the audio-only gen~ operator.) This is an excellent tutorial, so make sure you work through all of it. Continue your reading from “Jitter Operators” through to the end of the article. If your brain is a little tired, that’s just what it feels like to be advanced. Be sure to get enough sleep.

Help > Examples > jitter-examples > gen > jit.gl.pix.pinch.maxpat includes a simple codebox demonstrating function definitions, and well as a code version of the sample operator, including an attribute. For a glimpse of just how spectacular code can be, open Help > Examples > jitter-examples > gen > julia.quat.raytracer.maxpat. Even though jit.gl.pix is outputting a 2D image, the code inside makes the image look 3D.

A note about nomenclature: If you’ve already done some work with the vexpr or jit.expr objects, the conventions for inlet names might be a bit confusing. Open the “inlets_and_spaces” tab of jit.gl.pix.expr+codebox.maxhelp patch to see everything all laid out. While you're there, look through each of the other tabs to play with codebox’s functionality and modify it. It’s worth noting that using swizzle notation inside the expr and codebox operators works just like the swiz operator (i.e., “in1.r” instead of “swiz r”). Finally, note that “Param” must be upper-case inside codebox.

For a complete understanding of how the expr and codebox operators work, the GenExpr page is your final homework assignment. Put on your thinking cap, study each section carefully, then make a couple of simple patches with codebox.

And...you’re done. You may notice that it’s taken a couple of weeks to finish this article. (It might be time to do some laundry.) But now you’re an expert in building openGL fragment shaders with Max patches. The world is yours. Power to the pixel!

Check out the entire "My Favorite Object" series: bucket, stutter~, jstrigger, multislider, plot~, trigger, rate~, jit.slide, curve~, jit.gl.mesh, button, jit.gl.pix.

Learn More: See all the articles in this series