MaMe

All devices have been designed with Max. 

iquisme

Iquisme

Andrew Benson

After the RoboMoves recipe, I wanted to try a slightly more complex structure and work with a simple audio responsive motion control.  The repeated, branching structure of a tree provides the basis for this animation patch, and we use the amplitude of an incoming audio signal to scrub around the animation.  While this recipe follows much of the same structure as RoboMoves, the effect of this is really different.

First, we'll look at the structure of the animation nodes.  To create the tree, I created a repeated branching pattern with each parent node having 2 child nodes.  By scaling the each jit.anim.node, we get a structure where the children keep getting smaller as we go down the hierarchy.  The nice thing about this sort of repeated branching structure is that it was mostly just duplicating and connecting objects to get it set up.  If you wanted to make it even more complex, all you need to do is add in another layer or two of branching children.

To animate the tree, we use jit.path, just like in RoboMoves, but instead of a looping counter, we have a peakamp~ object driving the eval message.  The bline is there to smooth out the transitions and make it a little less hectic looking.


The branching structure of a tree provides the basis for this animation patch.

recipe-50-branching

Recipe 50: Branching

Kevin Heis

Muziko Kreito Universo is a three-dimensional visual and ambisonic sound installation. The work is experienced as a digital universe of intelligent musical creatures, where human participants act as gods controlling the environment as the creatures strive for auditory greatness. The work explores themes of creation and destruction, of power and servitude, and of the role of humans in an environment with other beings, as well as the fundamental relationship between the essence of sound and decision-making beings.

project-80-muziko-kreito-universo

Muziko Kreito Universo

In the last installment of the Video Processing System we left off with the beginnings of a basic live effects chain with basic compositing, blur, and color effects. Now that we've spent time building some basic interface wrappers for the jit.gl.slab object, it's time to start diving a little deeper. In this installment, we'll be working on some more advanced ninja tricks - creating the beginnings of a control/preset structure with assignable LFOs, and building a GPU-based video delay effect. These two parts will bring our system to a much more usable level, and allow for much more complex and interesting results. Ironically, most of what we are really doing in this installment is just an extension of bread-and-butter Max message passing stuff.

Download the patches used in this tutorial.

Download all the patches in the series updated for Max 7. Note: Some of the features in this system rely on the QuickTime engine. Using AVF or VIDDLL video engines will not support all functions. On both engines you may see errors related to loadram or have limited access to file record features.

The first thing we'll look at is the GPU video delay.  Everybody knows you can use the jit.matrixset object to store and recall matrix frames, but there is not a simple object solution for doing the same for OpenGL textures. Since textures (once they enter into the jit.gl.slab world) reside on your GPU's VRAM, and "readback" from the GPU to Jitter Matrix always comes at some performance cost, it often doesn't make any sense to convert back into a matrix just for one stage of processing. This also makes creating an efficient video delay in a slab-based effects network very difficult. Like any other difficult problem in Max there are a lot of different possible solutions. Here we will use texture capture - a technique that will open up a whole ton of possibilities once you understand how to do it.

Texture capturing means rendering OpenGL graphics directly to a texture, instead of drawing it onscreen. This captured imagery can then be used as a texture to render other things in your scene. You can use the @capture attribute on any OpenGL object to define a texture to capture to. This allows you to do things like create multi-pass rendering chains and is actually how the jit.gl.slab object works internally.

Let's go ahead and open the "texture-buffer" subpatch.  Inside that patch, you will see three poly~ objects loaded with the abstractions "vps.texset" and "vps.texlut".  The "vps.texset" abstraction just has a jit.gl.texture object and some gadgets to name and set the dimensions of that texture. This will act as our texture buffer. The important thing is that each of the textures loaded up in this poly~ is named something predictable so that it can be accessed elsewhere. The "vps.texlut" patch is the all-purpose reading and writing mechanism for the texture buffer, depending on the arguments. By relying on poly~ "target" messages, and knowing the attributes and messages that I need to send to the involved objects, we can easily patch together a very simple interface for the texture buffer.

The way the delay works is that each frame that comes in gets applied to the jit.gl.videoplane object, which is set to capture to a texture inside of "vps.texset". Each frame decrements a counter that sets which texture is captured to using "vps.texlut". We then add the frame delay amount and pass a delayed frame out of the outlet via "vps.texlut" sending "jit_gl_texure" messages. The modulo operator makes the delay patch function like a circular buffer, similar to tapin~/tapout~. To complete the effect, we have a simplified version of our compositing abstraction that allows us to combine the delayed version with the current frame. This is setup as a simple composited feedback network. More effects could be added to the feedback loop to create more extreme and complex looks.

Still with me? Good. Now, you could spend the rest of your days coming up with cool new video effects, but one of the most powerful tools of both video and audio performance is the ability to change things over time in interesting ways. Creating a robust structure to control your video system will allow you to do more than just apply a series of filters. So how do we create this structure? For starters I would like to review the fact that from the beginning we've been very nitpicky about maintaining a standardized interface to our modules (0.-1. values). This is going to make it a lot easier to create generalized controls for these values. At the center of our control system is the object pattrstorage. This object acts as our hub of communication with the UI objects in the patch and allows us to also save the state of the patch if we want to. In order to get our UI objects talking to pattrstorage, all we have to do is drop an autopattr in our patch and give each of our UI objects a Scripting Name. I recommend using meaningful and unique names for your controls, as you will want to be able to know the difference later.

Once we've got these items in place, you can easily "remote control" your UI objects by sending "objectname $1" messages to pattrstorage. The beauty of this is that there is no complicated send/receive network and state saving is also built in to the same system. With this in mind we can now create an assignable LFO using some simple Max tricks. If you peek inside the "menu-loader" sub-patch, you'll see that it's just some objects to send and receive messages and then parse the results. This subpatch sends a clear message to a umenu, queries the pattrstorage for the names of the controls in our patch with "getclientlist", which are then formatted so that a umenu can be loaded with the results. This gives us a menu full of available controls. Now, the cool part is, each time you add a new UI object and name it, the name will automatically pop up when you load the menu again. To generate the LFO waveforms, I just grabbed an abstraction I had lying around. The resulting values are then packed up with the object Scripting Name and sent to pattrstorage.   Note that audio must be on for the LFO to function.

Both of the modules we have added this installment are things that can be extended in many different directions by just using some Max techniques. The texture delay module could be extended to sequence delayed frames or play them randomly for more unpredictable motion, and any number of shivering and shaking can be introduced. The pattr-based control system can be used to map a physical controller to the UI, to generate any number of LFOs, or modulate things with other data sources. In our next installment, we'll revisit the movie-player and look at options for recording our output to disk.

In this installment, we'll be working on some more advanced ninja tricks - creating the beginnings of a control/preset structure with assignable LFOs, and building a GPU-based video delay effect. These two parts will bring our system to a much more usable level, and allow for much more complex and interesting results. Ironically, most of what we are really doing in this installment is just an extension of bread-and-butter Max message passing stuff. 

the-video-processing-system-part-3

The Video Processing System, Part 3

The second installment of Jitter Recipes is a collection of simple examples that began as weekly posts to the MaxMSP mailing list. Here you will find some clever solutions, advanced trans-coding techniques, groovy audio/visual toys, and basic building blocks for more complex processing. The majority of these recipes are specific implementations of a more general patching concept. As with any collection of recipes, you will want to take these basic techniques and personalize them for your own uses. I encourage you to take them all apart, add in your own touches and make these your own.

All the books in this series:  Book 1, Book 2, Book 3, Book 4 

Using jit.buffer~ to create simple audio solutions.

This example shows a couple of techniques that take advantage of the matrix interface of jit.buffer~.

On the left half we are recording a live signal into the upper buffer. This buffer is then output as a matrix, and fed through the jit.slide object to slowly fade between buffer states. This faded buffer-matrix is then loaded into the second jit.buffer~, which is what we will use for playback 

On the right half, we use a multislider and jit.fill to provide an easy drawing interface for waveshaping. Inside of the "use exprfill" subpatch, we've provided several common expressions that can be used to fill a jit.matrix with a windowing curve. This is particularly useful for doing granular synthesis.

Recording gestural parameter data over time

When recording high resolution videos from Jitter, it is often very difficult to get acceptable results from real-time manipulation of parameters. For this reason, it is often helpful to record real-time parameter changes on a frame-by-frame basis, and then play back these parameters at a rate that jitter can keep up with.

On the left half, we are using the pattrsorage object to save the state of all the named UI objects for each "frame". The benefit of this implementation is the ease and robustness of the pattr-family of objects as well as the ability to save the state of objects such as matrixctrl. 

The right half uses a matrix to store our parameter values, and then records frames into a jit.qt.record object.

Note that we're using jit.coerce to force our float32 matrix into an acceptable format for QuickTime (char).

Using Matrix feedback and distortion to create movement

This example is functionally very similar to the VideoSynth1 patch, except that we're using jit.repos instead of jit.rota to do the image distortion.

We are using jit.poke~ to perform automatic drawing on a named matrix (see VideoSynth1) 

This matrix is then distorted using jit.repos and then fed back to be drawn again using our named matrix.

To generate a constantly shifting repos-matrix, we've implemented a similar tactic as that used for the drawing (see bot2 subpatch). This matrix is then mixed with a matrix of normalized coordinates generated by the jit.expr object. This has the effect of gently repositioning the matrix at each frame, thus giving us smooth movement.

The output of jit.repos is then mirrored by sending it to jit.glue and inverting the x-dimension of the right input using jit.dimmap.

Using the stroke messages to jit.gl.sketch to render curves in OpenGL

Using a matrix to generate sketch commands for procedural drawing

With the introduction of JSUI, "stroke" commands were exposed for generating spline-curves. While this is pretty well-documented in the JavaScriptinMax PDF, many people aren't aware that you can also use these commands with jit.gl.sketch to do curvy line drawings.

The movement of the points is generated by adding a matrix of motion-vectors (see ParticleRave-a-delic) 

This matrix of point-locations is then sent to jit.iter to be broken down into 3-part lists.

For every matrix that is sent, we reset our sketch object and then begin our stroke using the "beginstroke line" message, and specifying the order (or smoothness of our curve) as well as color and width.

Each point of the matrix is then sent using the "strokepoint...."

Using expressions to generate values across a matrix

Reshaping a NURBS by setting control-point coordinates

Sometimes, when learning the ins and outs of OpenGL, it helps to be able to see just how this matrix of vertices is affecting our geometry. This example uses jit.cellblock to provide a visual interface to our matrix data. This allows us to perform precise distortions of our surface, which can be very useful for generating interesting graphics.

Using the "exprfill" message, we load our first two planes with signed-normalized values across the x and y dimension respectively. This gives us values between -1. and 1. By filling these matrices in such a way, we create a plane across the x and y coordinates of our GL-world. 

The z-plane is left empty for us to fill in as desired to create spatial distortion.

All of these matrices are then packed into one 3-plane matrix and sent on as the "ctlmatrix" for jit.gl.nurbs

We use jit.gl.slab to map a movie onto our jit.gl.nurbs shape. We could also load a shader program into this object to do processing on our texture.

Creating one's own particle system using matrix operations

Rendering multiple copies of the same geometry

Manipulating text using matrix operations

This patch was published for Xmas 2005, which should explain its holiday aesthetic. A particle matrix is generated which is then rendered as floating text-strings as well as vertices of a jit.gl.mesh. The text is also messed with a bit using some matrix processing.

The majority of these recipes are specific implementations of a more general patching concept. As with any collection of recipes, you will want to take these basic techniques and personalize them for your own uses.

jitter-recipes-book-2

Jitter Recipes: Book 2, Recipes 14-25

Jazer Giles

The orbits of the closest Near Earth Objects for the coming week are plotted in 3D. I use javascript (jweb) to query the NASA Jet Propulsion Laboratory's website to obtain the orbital coefficients. The orbital equation is solved in jit.gen. The audio is created by using an object's position to drive a spectral subtractive synth. The complete project files can be downloaded from jazergiles.com/projects. 

The orbits of the closest Near Earth Objects for the coming week are plotted in 3D. I use javascript (jweb) to query the NASA Jet Propulsion

near-earth-object-3d-orbital-simulation

Near Earth Object 3D Orbital Simulation

scn9a

MAX/Jitter was used for all aspects of the projection system, including mapping capabilities, visual effects and audio processing.

liste-noire-afire-afire

Liste Noire - "AFire, Afire"

Rnul interactive

The system was entirely made with Max. A lot of OpenGL layers could be controlled individually by the visitors. The system ran on a 8 core Mac Pro, each processor had it own patch to get the maximum power of the computer. The patches communicated through UDP. 

interactive-facade-projection-at-the-culture-night-breda

Interactive facade projection at the Culture Night Breda

oli larkin

Thijs Koerselman

Author: ICSRiM, University of LeedsThe i-Maestro 3D Augmented Mirror is a software tool built with Max and Jitter, designed to support bowed-string instrument training by using multimodal feedback. A realtime interface to a VICON 3D motion capture system allows the recording and display of bowing technique and posture in a 3D environment, and the extraction of a variety of pertinent features which can be monitored using visual or auditory displays. The 3D Augmented Mirror was developed as part of the i-Maestro EC-IST project at ICSRiM at the University of Leeds, UK.

project28-i-maestro-3d-augmented-mirror

i-Maestro 3D Augmented Mirror

Typically, when I talk to Jitter users about writing one's own shader programs for use with jit.gl.slab, I usually get glazed-over eyes and this sort of distant look of wonder. When I try to explain how easy it is, that look typically turns to one of annoyed disbelief. So, for a long time now I've been thinking about writing an article to de-mystify the process of writing your own GLSL shaders, and to help everyone avoid some common frustrations.

Please  download the Max 4.6 legacy files and the updated Max 5 files used in this tutorial.

If you are one of those people who fear text-based programming like your worst enemy, have no fear. For most things you will want to do with GLSL, the programming is really pretty lightweight, and you don't have to do deal with things like memory allocation or dealing with multiple functions. A shader program can sometimes be as short as a couple of lines of code, depending on how complex your ideas are.

For this article, I am going to walk you through the first GLSL shader I wrote, which I call "lumagate". The purpose of the shader is to generate an alpha channel, or transparency map, for the incoming image by only showing a specific range of luminance. This was an effect I was using a great deal to produce arbitrarily shaped textures, and I figured I could save a lot of CPU overhead by doing the calculations on GPU. For a jitter patcher-based example of this effect, see the patch called "01Lumagate_Patch".

Now, before we dive into the specific GLSL code I used to achieve this effect, we're going to have a look at the JXS file format. JXS is an XML-based file format that jit.gl.slab and jit.gl.shader use to load shader code and configuration data (initializing variables, metadata etc.). The JXS format allows you to specify all of the expected parameters and their default values in a familiar and easily readable way. Go ahead and open the file "tut.blank.jxs" in a text-editing application. This is a very simple shader program that just multiplies the input with a variable and passes it to the output.

	shader program that does absolutely nothing!
	

At the very beginning of the file, you will see a tag that says jittershader name="blank". This line tells Jitter that the file is to be read as a shader file, and that its name is "blank". You can also describe the shader itself using the description tag.

The param tag is the most important one for you to understand.  This is where you define any uniform parameters that you are using in your code.  A uniform parameter is a value that can be passed into the shader from the patcher, but can't be altered by the shader program itself. These will typically be used for any value that you need an interactive interface for. Within the param you must specify the name of your variable, the data type (I'll get to that later), and the default value for the parameter. The default value is passed to the shader program automatically when it is loaded. Think of it like a [loadbang]. You might notice that the textures used by the [jit.gl.slab] object are also defined using the param tag.  In the case of textures, the default value will be correspond to the [jit.gl.slab] input, starting with "0".

Once you create the parameters of your shader program using the param tag, you then need to bind them to either the fragment or vertex program. For the sake of this article, we're only going to look at the fragment program, which is where all of the per-pixel processing is done in GLSL. The bind tag tells Jitter where to send the parameter info. If you try to bind a parameter to a program that doesn't have a variable with the same name, you will get a compiler warning when you load the shader.

The program tag allows you to specify a vertex and a fragment program for your shader. Using the "source" argument, you can specify another file for a specific program. In this case we are using "sh.passthrudim.vp.glsl" vertex program, which is one of several handy little utilities in the "jitter-shaders/shared" folder. This particular one just does some basic setup work for you and passes some useful variables to the fragment program.

I would recommend to keep a copy of this "tut.blank.jxs" on hand to use as a template for any shaders that you want to write in the future. Now that we've covered the file format, let's get into some real GLSL code!

The first variable that any worthwhile texture processing shader should have is the texture sample.  This is defined as

in our example code (for non-rectangular textures use sampler2D()). This means that we are using a texture with the name "tex0" that will be sampled and processed by our program. To sample multiple pixels to be processed together (such as in a blur filter), you will need to use a sampler for each one. The way that fragment shaders work, you are essentially programming a process for a single output pixel of the framebuffer. This is then run for each pixel of the output texture in a massively parallel way. Because of the architecture of the graphics card, you only have access to the single framebuffer output pixel within your program.Our next variable is a varying variable, which means that it includes data that is being sent from the vertex program. The "texcoord0" varying variable represents the current coordinates of the texture pixel that you are processing. Combined, these two variables give you the data needed to access your texture and process it within the shader program.

The other variable here is a uniform variable called nothing that represents a floating-point number that is passed from the interface to MaxMSP to your shader. The beginning of the shader program is represented by void main() {, which will be familiar to anyone who has written code for Java or C. For those who are unfamiliar with programming, this basically just tells the compiler that what follows is the main function of our program.

In addition to familiar data types such as int and float, GLSL also uses a number of special data types that are optimized to work with video and OpenGL data. The most important of these are "vectors", which are small collections of numbers with 2,3, or 4 members. These are declared as:

Color in GLSL is almost always represented as a vec4 value (red,green,blue,alpha). To convert a single float value to a vec4 value, you can use the constructor vec4(floatvalue).  You can also convert between different types of vectors using the constructor:  vec4(ivec4variable).  This is often necessary as GLSL will rarely allow you to mix and match data types within an equation or function.Another really cool thing you can do with vectors in GLSL is to use the "swizzle" notation to reorder and create new vectors. For example, if you wanted a vec4 variable with the red and green values swapped, you can write it as -

Notice that you can use either cartesian coordinate or color name swizzle, but you can't mix and match.  This will not work:

If you would like to learn more about the GLSL data types and special ways to deal with them, I strongly encourage you to read the official GLSL documentation.

What follows is a ridiculously simple GLSL program:

void main()
{   
	gl_FragColor = texture2DRect(tex0, texcoord0)*vec4(nothing);
}

Here we are outputting the result of multiplying the value of the current pixel by our nothing value.  The variable gl_FragColor is a GLSL built-in variable that defines what color a specific pixel should be set to. Think of this as the output of your program. Usually, this will be the last part of the process.In this statement, you will also notice the function texture2DRect(). This function requires two values - a specified texture to sample and the vec2(x,y) coordinates of the texture sample. All of our texture-displacement shaders do their work by playing with the texture coordinate values passed to the texture2DRect() function. The result of this function is a vec4 value - a collection of 4 floating-point numbers - that represents the color of that pixel as (red,green,blue,alpha). You will notice that after the "*" (multiplier) there is the vec4(nothing) statement.  This converts the float variable nothing into a vec4 variable for the purpose of multiplying with our texture color. You need to be very explicit in converting data to the proper format.

This concludes our detailed walk-through of a shader program.  Are you still with us?

Now we are going to take this lame shader program and turn it into something really useful and awesome. Have a look at the file "01LumagatePatch" for a patcher-based description of the functionality that we are trying to create with this shader. In order to get this working in GLSL, it will help to break down the steps necessary for it to work: 

Compare luminance to our high and low thresholds

Multiply our two comparisons together to get a combined range

Use the value of the combined comparison as the value of the Alpha, or transparency, channel

Now that we know what we need to do, all of this is pretty simple to implement in GLSL code.First, we need to get the value of the pixel into the shader program.  As you saw earlier, this is done using the texture2DRect() function.  You will need to add the following line to the beginning of your main() function:

This creates a new vec4 variable "a" and fills it with the color value of the current pixel.Now that we have the color value, we can begin to do some calculations. To convert the RGB value to a luminance value, we will borrow a technique used in "co.lumakey.jxs", which is to do a dot-product of the RGB value with a set of luminance coefficients (0.299,0.587,0.114,0.). Rather than just writing these values directly in our program, it is generally far more readable to create a meaningful name for this set of numbers. To do this, we will define our luminance coefficients as a "const" (constant value) variable:

const vec4 lumcoeff = vec4(0.299,0.587,0.114,0.);

We do this before the beginning of the main function because all of the varying, uniform, and const variables must be declared as global variables. Now we can add the following line after our texture lookup statement:

This creates a new float variable "luminance" which represents the dot-product of "a" and "lumcoeff". A dot-product returns a float value which comes from multiplying each value in the first vector by the associated value in the second vector, and then adding all the members of the resulting vector together (this.r*that.r+this.g*that.g...). You may be thinking, "Why not just use a mathematical expression for this?" The answer is that whenever possible, it is best to use built-in GLSL vector function as these are optimized for the best performance. In fact, whenever possible, using vec4 calculations is going to give you the best efficiency as GLSL is optimized for vec4 data types.

void main()
{   

	vec4 a = texture2DRect(tex0, texcoord0);

	// calculate our luminance
	float luminance = dot(a,lumcoeff);

	// compare to the thresholds
	float clo = step(lum.x,luminance); 
	float chi = step(luminance,lum.y);

Now that we have the luminance value, we will look at the step() function to do our comparison operations. This GLSL built-in function returns a "1." if the second value is greater than the first (edge) value, and a "0." if it is less than or equal to the edge value. Knowing this, we can create a vec2 variable - a collection of 2 floats - that represent our high and low luminance thresholds. As shown previously, there is a little bit of setup involved in creating an interactive variable for a shader. First we use the param and bind tags to define the parameter and bind it to our fragment program. To see how we do this, take a peek at the file "tut.lumagate.jxs". I set the default values to (0.,1.) to have the visible range include all possible values by default when the shader is loaded. We then define our uniform vec2 lum; in the section just before the beginning of the main() function.

Now that we have a variable representing the luminance thresholds, we can perform some comparison operations. First we will compare with the low threshold to see if our luminance is greater than the minimum:

Notice that I used lum.x to represent the first value of the vec2 variable lum. We do a similar thing to compare for the upper threshold, but we reverse the terms since we want to do a less-than comparison:

Now all we need to do is combine the results of the two comparisons and load this into the Alpha-channel of the output color. To combine the two comparisons, we will just multiply them together:

Now we have a variable "amask" that will become the alpha channel of our pixel when we include the following line:

Note that in order to accomplish this we used the vec4() constructor, using the first (rgb) values of the texture sample a.rgb and amask as the Alpha value. Tricky, right? Don't forget to "Save As..." and name your shader using the .JXS extension. Make sure that your text-editor doesn't try to add any extra extensions to the file.

void main()
{   

	vec4 a = texture2DRect(tex0, texcoord0);

	// calculate our luminance
	float luminance = dot(a,lumcoeff);

	// compare to the thresholds
	float clo = step(lum.x,luminance); 
	float chi = step(luminance,lum.y);

	//combine the comparisons
	float amask = clo * chi;

	// output texture with alpha-mask
	gl_FragColor = vec4(a.rgb,amask);
}

This leaves us with a great shader program that will only show us a texture pixel (texel in official terms) if it falls within our defined range of luminance values. To test this shader, open up the file "03Shader_Test" in MaxMSP/Jitter and load a movie file. Try reading your saved shader into jit.gl.slab. Are you getting any compiler errors? How did you do?

This concludes your first lesson in mastering the new world of GLSL shaders. If you want to learn more, I highly recommend poking around the "jitter-shaders" folder and grabbing the official GLSL specification(PDF) or the GLSL Orange Book. Now that you have a basic understanding of how shaders are written, you should be able to work out how we are doing things in other shaders. If you come across a built-in function you don't recognize, just check the official GLSL documentation. This is really just a small taste of the cool things you can do with GPU-based video processing. When I get some time I will try and write up some more tips and tricks, but this should keep you busy for a little bit. If you have any questions, I hope that you post them to the Jitter forum.

Search Results