The purpose of this chapter is to fill in the details of what a Matrix Operator is and how it works.
Matrix data in Jitter is typically considered raw data without respect to what the data represents. This permits simple fundamental operations to be applied to different sorts of data without needing to know any special information. For this reason most MOPs are general purpose. The jit.scalebias example from the preceding chapter could be considered video specific in its terminology, and type and plane count restrictions, but fundamentally it is just calculating a product and sum on each plane of an incoming matrix. In this chapter, we'll cover the details of how to configure MOP inputs and outputs, any attribute restrictions or linking for those inputs and outputs, what you must do in your matrix_calc method and how you expose your MOP to the Max environment, overriding default behavior if necessary.
As discussed in the Matrix Operator Quick Start, for MOPs you must create an instance of jit_mop with the jit_object_new() function and add it to your Jitter class as an adornment with the jit_class_addadornment() function. The jit_mop object holds information such as how many inputs and outputs the object has, what types, plane count, and dimension counts are supported, and how inputs should respond to incoming matrices. This information is only relevant to wrappers of the Jitter object which actually maintain additional matrices for inputs and outputs, as is the case with the MOP Max wrapper class. When used from C, Java, or JavaScript, it is the the programmer's responsibility to pass in matrices that conform to any restrictions imposed by the MOP. An example of instantiating and adding the jit_mop object is below.
Each instance of jit_mop contains some number of inputs and outputs, specified by the input and output arguments to the constructor. For each of these inputs and outputs there is an instance of jit_mop_io which records information specific to that input or output, such as type, plane count, and dimension restrictions. You can access the input or output objects by calling the getinput or getoutput methods with an integer index argument as below:
Once you have obtained references to these inputs or outputs, you may query or set the jit_mop_io attributes. The attributes typically configured are: types, which is a list of symbols of permitted types, the first of which being the default; mindim and maxdim, which are the minimum and maximum permitted sizes for each dimension; mindimcount and maxdimcount, which are the minimum and maximum permitted number of dimensions permitted; minplanecount and maxplanecount, which are the minimum and maximum number of planes permitted; typelink, which is the flag that determines if the I/O should change its type to whatever the leftmost incoming matrix is; dimlink, which is the flag that determines if the I/O should change its dimensions to whatever the leftmost incoming matrix is; and planelink, which is the flag that determines if the I/O should change its plane count to whatever the leftmost incoming matrix is.
By default, all types, dimensions and plane count are permitted, and all linking is enabled. If you wish your MOP to have some specific restrictions, or difference in linking behaviors for any input or output in particular, you can set the corresponding attributes. For example, to set the plane count to always be four planes, you would set both the minplanecount and maxplanecount attributes to 4, as below:
The jit.scalebias example could have set the planecount using the minplanecount and maxplanecount attributes rather than calling the utility function jit_mop_single_planecount(), which internally sets these attributes. A similar thing could be done to restrict type and dimensions. As for linking, if you wish to develop an object where the right hand input does not adapt to the size of the leftmost input, as is the case with jit.convolve, you would turn off the dimlink attribute, as below:
Similar could be done to remove type and planecount linking, and the utility functions jit_mop_input_nolink() and jit_mop_output_nolink() set all of these link attributes to false (zero).
For right hand matrix inputs, incoming data is typically copied by the MOP Max wrapper class. When an incoming matrix is received by the MOP Max wrapper class, a function called the ioproc is called, and the default ioproc copies the data, using the current input attributes (which might be linked to the lefthand input). The default ioproc can be overridden by calling the ioproc method followed by a function with the signature as listed below in the jit_mop_ioproc_copy_adapt() function. The jit_mop_ioproc_copy_adapt() function will always adapt to that inlet's incoming matrix attributes, as long as they don't conflict with any restrictions. The SDK project for jit.concat demonstrates the use of the jit_mop_ioproc_copy_adapt() function.
You can specify variable input/output MOPs with a negative argument for input and/or outputs when constructing your jit_mop object. When the using variable inputs and/or outputs, there is not a jit_mop_io for each input and/or output within your class definition, and therefore the template type, dim, planecount, and linking attributes are not settable. If anything but the default behavior is required, you must accomplished it in another way — for example, either by overriding the jit_matrix method of the MOP Max wrapper class, or defining an mproc method to be called from within the standard jit_matrix method of the MOP Max wrapper class. The jit.pack, jit.unpack, jit.scissors, and jit.glue objects are a few SDK examples of MOPs with variable inputs and outputs. More information on overriding the jit_matrix, mproc, and other default methods of the MOP Max wrapper class is covered later in this chapter.
Once you have configured all of the inputs and outputs of your jit_mop object, you must add your jit_mop object to your Jitter class with the jit_class_addadornment() function. Adorments can be queried from the Jitter class at any time by calling jit_class_adornment_get() with the Jitter class pointer and the class name of the adornment object, as demonstrated below.
The entry point of the MOP Jitter class is the matrix_calc method, which is passed a list of matrices for the input, and a list of matrices for the output. It is not the responsibility of the matrix_calc method to perform any copying and adaptation behavior, but rather simply ensure that the matrices are valid, compatible, and if so, process. Certain objects may modify the dim, type, or planecount of the output matrices — e.g. the SDK project, jit.thin. However, it is the calling party's responsibility to perform any copying and conformance to MOP I/O restrictions as defined by the jit_mop_io objects—i.e. either the Max wrapper class, or the C, Java, or Javascript code which calls the matrix_calc method.
The input and output lists passed as arguments to your matrix_calc method are Jitter objects, and pointers to the individual inputs and outputs are acquired by calling the getindex method with an integer argument specifying the zero based list index. The return values should be tested to make sure they are not null. For example:
Technically, you can also pass in an instance of jit_matrix in place of a list for the input or output arguments, since jit_matrix has a getindex method which returns the jit_matrix instance. This is an example of dynamic binding at work. Another example of dynamic binding inside the matrix_calc method is that the list elements might be instances of jit_mop_io, rather than instances of jit_matrix. However, since Jitter uses dynamic binding and the jit_mop_io object is a "decorator" class for jit_matrix, all corresponding methods are passed on to the jit_matrix referenced by the jit_mop_io. In fact, any Jitter objects which respond to the standard interface for jit_matrix could be passed as inputs or outputs. If this seems confusing, you need not think about the underlying implementation further, but instead can assume that what is being passed in is simply an instance of jit_matrix. After all it should behave like one, even if it is not.
Prior to working with a matrix, it is necessary to "lock" it so that the data and attributes will not be changed across the duration of the operation. This is accomplished by calling the jit_matrix instance's lock method with an integer argument of 1 (true) to lock the matrix. You should store the current lock state to restore when you're done processing. The lock operation should be the first thing to do after ensuring that the matrix objects are not NULL. For example
Once you have locked the matrices, you are ready to find out some information about them. This is accomplished by calling the getinfo method with a pointer to an instance of the t_jit_matrix_info struct. The t_jit_matrix_info struct contains several common attributes of the matrix and data organization of the matrix data, and is a useful way to obtain this information in one call, rather than querying each attribute individually. This information is typically tested to verify compatibility with any assumptions the matrix_calc method needs to make (since this method might be called from C, Java, or Javascript, you cannot assume that the MOP Max wrapper will have enforced these assumptions). It is also used to perform the appropriate pointer arithmetic based on type, plane count, dimensions, and the byte stride of those dimensions, since higher dimensions may not be tightly packed. The t_jit_matrix_info struct is listed below:
And here is an example of calling the getinfo method to fill out the t_jit_matrix_info struct:
The t_jit_matrix_info struct is the meta data, but the actual matrix data can be accessed by acquiring the data pointer. You accomplish this by calling the matrix's getdata method, passing in a pointer to a pointer. This pointer can be any type, but it is typically a char (or byte) pointer since you may need to perform bytewise pointer arithmetic depending on the type and dimstride of your matrix. It is essential to verify that this pointer is valid before attempting to operate on the data, as demonstrated below.
While it is possible to incorporate the data processing code inside the matrix_calc method, it is typical to rely on other routines to accomplish the N dimensional processing through recursion, potentially dispatching to multiple processors. The N-dimensional recursive processing function (typically named myobject_calculate_ndim) is discussed in the next section. You should pass in to the calculate_ndim function your object pointer, the overall dimension count, dimension sizes, planecount to consider in your calculation, together with the necessary matrix info structs and data pointers for each input and output. You can call this method directly as is the case in the following code:
Or you can call this method with the parallel processing utility functions provided with Jitter 1.5 to automatically dispatch the processing of large matrices across multiple processors when available. This figure illustrates the dispatching and calculating of the parallel processing utility:
The parallel processing is accomplished by breaking up the matrix into smaller matrices that each reference subregions of the original inputs and outputs. No new objects are created, but rather just additional t_jit_matrix_info structs and offset data pointers. Jitter 1.5 maintains a pool of worker threads for this purpose, so there is no thread creation overhead, but rather only some small thread synchronization overhead. Jitter 1.5 only dispatches across multiple threads when the data count is large enough to justify this thread synchronization overhead.
An important thing worth noting is that if your object performs some kind of spatial operation (e.g. convolution, rotation, scaling, etc.), you will either need to account for the matrix segmentation used by the parallel utilities or avoid using parallel processing and call directly in the current thread. Since the jit.scalebias example only processes one pixel at a time (i.e. a pointwise operation), it is inherently parallelizable, so it takes advantage of multiple processors as below:
Important Note: If you aren't sure if your object is a pointwise operator, or don't fully understand how to make your algorithm parallelizable, you shouldn't use the parallel utility functions in your object. You should simply call the function directly.
In the Matrix Operator Quick Start chapter, we discussed how to define a recursive function to process N-dimensional data in 2D slices, using the jit.scalebias object as an example. This example was restricted to processing four plane char data, but many Jitter objects work with any type of data and any plane count. In order to support all types and plane counts, there needs to be some case handling to know how to step through the data, and what type data to interpret as so that you can perform the appropriate operations. There are a number of ways to approach this logic, and decisions to make with respect to optimization. All this case handling can be a bit cumbersome, so when initially developing objects, it probably makes sense for you to focus on a single type and plane count, and only after you've adequately defined your operation, attempt to make your code robust to process any type of data and consider optimization of certain cases. The use of C macros, or C++ templates might be useful things to explore for better code re-use. As for code optimization, typically a decent atomic element to try and optimize is the "innermost" loop, avoiding branch conditions where possible.
This function is at the heart of the logic you will add in your own custom object. Since there is no "right way" to process this data, we won't cover any more code listings for the recursive N-dimensional processing function. However, the SDK projects that are good examples include: jit.clip, which performs a planar independent, pointwise operation (limiting numbers to some specified range); jit.rgb2luma, which performs a planar dependent, pointwise operation (converting RGB color to luminance); and jit.transpose, which performs a planar independent, spatial operation (rows become columns). For more ideas about N-dimensional matrix processing, we would recommend reading one of the several books available on 2D signal processing and/or image processing. Most of these concepts are easily generalized to higher dimensions.
MOP Max wrapper classes typically have a large amount of default behavior, as setup through the max_jit_classex_mop_wrap function, based on the jit_mop Jitter class adornment, and user specified flags. You can either override all of the default behavior or just specific features. If you wish to override all of the default behavior, you can use the flag MAX_JIT_MOP_FLAGS_OWN_ALL, when calling the max_jit_classex_mop_wrap() function. If you need to make use of the jit_mop adornment(), the jit_mop can be looked up by calling the jit_class_adornment_get() method on the Jitter class. The jit_mop_io inputs and outputs can be queried and their attributes inspected, similar to how they were set in the MOP Jitter class definition, described earlier in this chapter. Here is an example of how to look up the jit_mop adornment of the jit.scalebias object:
By default, a jit_matrix method is added which automatically manages matrix copying and calculation based on the incoming data. Most typical MOPs simply use the default jit_matrix method. However there are instances where it is necessary to override the default MOP method to get special behavior, such as recording which matrix input data is being input to as is the case for the jit.op SDK example, or to do something other than standard copying and adaptation as is the case for the jit.pack or jit.str.op SDK examples, or to prevent any jit_matrix method at all, as is the case for the jit.noise SDK example. To prevent the default jit_matrix method from being defined, you can use the flag MAX_JIT_MOP_FLAGS_OWN_JIT_MATRIX, when calling the max_jit_classex_mop_wrap() function. To define your own jit_matrix method, you can add an A_GIMME method bound to the symbol jit_matrix, in your ext_main() function. Here's an example from jit.op:
The jit.pack and jit.str.op examples are a bit more involved and also better illustrate the kinds of tasks the default jit_matrix method performs.
A MOP Max wrapper class typically has a bang and outputmatrix method. These two methods are typically equivalent, and by default, both send out the most recently calcuated matrix output. Certain objects that don't have a matrix output, like the jit.3m SDK example, typcially override these messages with their own bang and sometimes outputmatrix method. These methods can be overridden by using the MAX_JIT_MOP_FLAGS_OWN_BANG and MAX_JIT_MOP_FLAGS_OWN_OUTPUTMATRIX flags when calling the max_jit_classex_mop_wrap() function. These flags are typically both passed in together.
For each input and output, other than the leftmost input, there is, by default, an attribute added to query and set that input or output's matrix attributes, including name, type, dim, and planecount. While overriding the default attribute behavior is conceivably necessary to perform very specialized behavior, it is not used by any of the SDK examples. To prevent the addition of the default attributes for name, type, dim, and planecount, you can use the MAX_JIT_MOP_FLAGS_OWN_NAME, MAX_JIT_MOP_FLAGS_OWN_TYPE, MAX_JIT_MOP_FLAGS_OWN_DIM, and MAX_JIT_MOP_FLAGS_OWN_PLANECOUNT flags when calling the max_jit_classex_mop_wrap() function. To define your own attributes, you would follow the same means of defining any attributes for a Max wrapper class with the appropriate attribute name you wish to override.
By default, a clear and a notify method are added. The default clear method clears each of the input and output matrices. The default notify method, max_jit_mop_notify(), is called whenever any of the matrices maintained by the MOP are changed. If it is necessary to respond to additional notifications, it is important to call the max_jit_mop_notify function so that the MOP can perform any necessary maintenance with respect to input and output matrices, as demonstrated by the jit.notify SDK example. These methods can be overridden using the MAX_JIT_MOP_FLAGS_OWN_CLEAR and MAX_JIT_MOP_FLAGS_OWN_NOTIFY flags, respectively, when calling the max_jit_classex_mop_wrap() function. Object registration and notification is covered in detail in a future chapter, but the jit.notify notify method is provided as an example.
By default, adapt and outputmode attributes are added to the MOP Max Wrapper. These attributes determine whether or not to adapt to incoming matrix attributes, and whether or not the output should calculate a new output matrix, output the last calculated matrix (freeze), pass on the input matrix (bypass). To prevent the addition of the default attributes for adapt and outputmode, you can use the MAX_JIT_MOP_FLAGS_OWN_ADAPT, and MAX_JIT_MOP_FLAGS_OWN_OUTPUTMODE flags when calling the max_jit_classex_mop_wrap() function. To define your own attributes, you would follow the same means of defining any attributes for a Max wrapper class with the appropriate attribute name you wish to override.
For many types of operations, it's not required to fully override the default jit_matrix method and any adaptation. If your object simply needs to override the way in which the Jitter class' matrix_calc method and outlet functions are called, you can do so by defining an mproc method, which will be called instead of the default behavior. The jit.3m SDK project is an example where after it calls the Jitter class' matrix_calc method, it queries the Jitter class' attributes and outputs max messages rather than the default jit_matrix message output.
As we discussed in the Matrix Operator Quick Start, inside your Max class' constructor you need to allocate the matrices necessary for the MOP inputs and outputs, the corresponding matrix inlets and outlets, process matrix arguments and other MOP setup. And in your destructor, you need to free oup MOP resources. Typically you would accomplish this all with the standard max_jit_mop_setup_simple() and max_jit_mop_free() functions, however there are some instances where you may need to introduce custom behavior.
The max_jit_mop_setup_simple() function calls max_jit_mop_inputs() and max_jit_mop_outputs() to define any necessary proxy inlets, outlets, and internal matrices. The listing for these functions are provided below to illustrate the default behavior, and a few SDK projects we recommend investigating further are jit.scissors, jit.glue, jit.pack, and jit.unpack.
The max_jit_mop_setup_simple() function calls max_jit_mop_matrix_args() to read any matrix arguments, and if present send them to any linked inputs/outputs and disable the adapt attribute. The listing is provided below to illustrate the default behavior.
