andrea mancianti

I have been trying to do quite a simple thing in many different ways, but so far have not found any optimal strategy.

I would like to be able to iterate within a large-ish multidimensional/multiplanar jitter matrix to be able to compare and operate on a point by point level. I already tried java with copyMatrixToArray method and a double nested for loop, but above a 10x10, 3-planes matrix the performances are quite low. (below 20 fps on a new macbook pro).

In javascript I did not find any method to copy more than a single cell (getcell), which given the two-dimensional nature of the matrix ends up in 4 nested for loops... Also very very slow. Is there any equivalent to copyMatrixToArray?

I wouldn't mind trying out Lua, but the getrow function seems to return always only the first row of the referred matrix no matter what value you pass and all in all that part is quite under-documented. In general most of the topics related to scripting and jitter in the forum date back to at best 7 years ago and i have the impression that there is much less effort in that direction.

Gen's codebox would be the perfect place to go, but as far as I know it doesn't allow for loop or any other loop construct.

What do you think? Any suggestions or ideas?

Hi all!
I have been trying to do quite a simple thing in many different ways, but so far have not found any optimal strategy.
I would like to be able to iterate within a  large-ish multidimensional/multiplanar jitter matrix to be able to compare and operate on a point by point level.  I already tried  java with copyMatrixToArray method and a double nested for loop, but above a 10x10, 3-planes matrix the performances are quite low. (below 20 fps on a new macbook pro).
In javascript I did not find any method to copy more than a single cell (getcell), which given the two-dimensional nature of the matrix ends up in 4 nested for loops... Also very very slow. Is there any equivalent to copyMatrixToArray?
I wouldn't mind trying out Lua, but the getrow function seems to return always only the first row of the referred matrix no matter what value you pass and all in all that part is quite under-documented. In general most of the topics related to scripting and jitter in the forum date back to at best 7 years ago and i have the impression that there is much less effort in that direction.
Gen's codebox would be the perfect place to go, but as far as I know it doesn't allow for loop or any other loop construct.
What do you think? Any suggestions or ideas?
Thanks a lot!


iterate-in-a-jitter-matrix-and-scripting-in-jitter

What do you actually want to achieve? [jit.op] is quite capable for point operations.

I am actually trying to implement some drawing procedures based on the mutual relationships between all the points in a matrix. A very simple and classic example would be to test the distance between every single point with each other, and for instance draw a line (or do any other thing) between those points whose distance is below a threshold, in the style of many classic "Processing" sketches.

I am trying to experiment with approaches that are a little bit outside of the powerful parallel thinking paradigm of jitter, but admittedly it seems to hit performance walls very fast, actually much faster than within processing itself for example.

I was also interested in understanding, in a more general sense, where Max is going to in relation to scripting in general, as it seems to me that the enthusiasm of some 5/10 years ago is not so high anymore.

Hi Jean-Francois!
I am actually trying to implement some drawing procedures based on the mutual relationships between all the points in a matrix. A very simple and classic example would be to test the distance between every single point with each other, and for instance draw a line (or do any other thing) between those points whose distance is below a threshold, in the style of many classic "Processing" sketches.
I am trying to experiment with approaches that are a little bit outside of the powerful parallel thinking paradigm of jitter, but admittedly it seems to hit performance walls very fast, actually much faster than within processing itself for example.
I was also interested in understanding, in a more general sense, where Max is going to in relation to scripting in general, as it seems to me that the enthusiasm of some 5/10 years ago is not so high anymore.
Thank you anyway for your answer!


I just want to say that the codebox in gen does have for loops, and other loops, so you can do what you want to achieve in the gen objects. See the "GenExpr" article in the documentation for details.

I'm currently trying to do exactly what you're going for (and have been attempting it for a good while), as well as Newton forces for all particles in a matrix, but making those loops do what I want is hard, since I'm not used to "regular" coding at all, and my skill at doing coordinate operations inside jit.gen/jit.gl.pix is lacking. If anyone has some examples of iterating through all cells of a matrix in jit.gen/jit.gl.pix for every cell, I'd love to see it. I feel like the solution is very simple, I've just been banging my head for a long time. Attaching one of my own sluggish attempts at codebox, maybe someone can point me in the right direction..

Also, Federico Foderaro said in a facebook post that using the codebox for-loops to calculate relations between all cells in a matrix is not working well for big matrices. Obviously it's gonna be a lot of calculations if you crank up the dimensions, but does anyone know if there is any reason other than that for loops in gen to be inefficient?

I just want to say that the codebox in gen does have for loops, and other loops, so you can do what you want to achieve in the gen objects. See the "GenExpr" article in the documentation for details. 

I'm currently trying to do exactly what you're going for (and have been attempting it for a good while), as well as Newton forces for all particles in a matrix, but making those loops do what I want is hard, since I'm not used to "regular" coding at all, and my skill at doing coordinate operations inside jit.gen/jit.gl.pix is lacking. If anyone has some examples of iterating through all cells of a matrix in jit.gen/jit.gl.pix for every cell, I'd love to see it. I feel like the solution is very simple, I've just been banging my head for a long time. Attaching one of my own sluggish attempts at codebox, maybe someone can point me in the right direction.. 



Also, Federico Foderaro said in a facebook post that using the codebox for-loops to calculate relations between all cells in a matrix is not working well for big matrices. Obviously it's gonna be a lot of calculations if you crank up the dimensions, but does anyone know if there is any reason other than that for loops in gen to be inefficient? 

Yes, Gen Expr does have a for loop, but the problem is not as trivial as it might seems. To achieve what I (we?) want to do, you need basically to be able to do two things:

on the one hand to sample both the matrices, that, with some oddities, should be achievable with something along the line of the following patch:

where the parameter select is our index of the for loop. This has to be performed "for every cell of the first matrix for all the cells of the second one", which in psudocode is something like this:

then calculate the distance between all the points with for ex this formula:

distance=sqrt(pow((array[i].x-array[j].x), 2) + pow((array[i].y-array[j].y), 2) + pow((array[i].z-array[j].z), 2));

Now, if the condition is satisfied copy the value of the matrix to store. This second part is, to my understanding impossible to produce in gen, because there is no array notion and thus you can't just tell the program to store your value here or there. 

In addition, performing the two for loops as shown, produces an output matrix of dimentions equal to the power two of the dimentions of the input matrix. (There would be tons of possible optimisation tricks if it would be, let's say Java, where the above code would take an hour to write, but in gen I cannot do it, I just don't find a way out). 

So to conclude, yes, if someone has done this please help us! :)

Yes, Gen Expr does have a for loop, but the problem is not as trivial as it might seems. To achieve what I (we?) want to do, you need basically to be able to do two things: 

on the one hand to sample both the matrices, that, with some oddities, should be achievable with something along the line of the following patch: 

where the parameter select is our index of the for loop. This has to be performed "for every cell of the first matrix for all the cells of the second one", which in psudocode is something like this: 

    for (int j=0;j<matrix.lenght;j++) {  

 then calculate the distance between all the points with for ex this formula:

        distance=sqrt(pow((array[i].x-array[j].x), 2) + pow((array[i].y-array[j].y), 2) + pow((array[i].z-array[j].z), 2)); 

Now, if the condition is satisfied copy the value of the matrix to store. This second part is, to my understanding impossible to produce in gen, because there is no array notion and thus you can't just tell the program to store your value here or there. 
In addition, performing the two for loops as shown, produces an output matrix of dimentions equal to the power two of the dimentions of the input matrix. (There would be tons of possible optimisation tricks if it would be, let's say Java, where the above code would take an hour to write, but in gen I cannot do it, I just don't find a way out). 
So to conclude, yes, if someone has done this please help us! :)

I'm new to Max but not to numerical computing, and I've had similar questions about how to implement arbitrary matrix operations in a reasonably performant way. One thing I haven't tried yet: would it work to (1) flatten input matrices to buffers / Gen data objects / whatever, (2) iterate over the flattened data in a gen~ object, computing indices manually (e.g. x*n_cols + y) and using peek()/poke(), (3) expand the result data to a matrix? Seems like it might need a little tweaking to avoid slow operations, and you're definitely on your own as far as design-related performance implications, but I don't see anything obviously wrong with it.

Now I might actually be curious enough to try this once I get some time.... Is there an obvious reason it wouldn't work?

(Also I have to admit I'm a little confused about why it seems to be common to store particles in NxNxC arrays when all you want is N^2 unrelated particles in C dimensions.... Does that make them easier to draw or debug than storing them in an N^2xC array?)

If you're this far down in the weeds and worrying about speed, I'd suggest that you look into writing a custom Max external for this kind of stuff. Have you considered that?

I am happy the conversation is getting hotter around here! :) Matt, I had the same thought about passing through gen~ but let's say I would be happy to discover there is some less hacked solution. In case there would be none I would definitely give it a try.

For what concerns the particle/arrays question, I have been wondering the same, (when I have done my java implementation of this code, I used "normal" 1 dimensional arrays) but i guess the reason lies in the architecture of jitter that basically uses matrices organised as pictures (n*m*planes, typically 4) as data repositories for all sort of data, and all the tools are operating with those representations in mind. Plus this way you can easily do "shaders style" displacements using other pictures.

Finally, for what Gregory suggests, i.e. writing an external, I agree that would be the most elegant and sustainable way to do it, it should not be too complicated as it is mostly about porting an already existing java external to C, but still the amount of stuff I still don't understand about the way c and max relate, together with some serious pointers/memory handling allergy, make me shiver.. Plus I found a book for learning how to write msp externals, but nothing about Jitter... But if there are other people interested, why not make some sort of study group! I'd be happy to!

I am happy the conversation is getting hotter around here! :) Matt, I had the same thought about passing through gen~ but let's say I would be happy to discover there is some less hacked solution. In case there would be none I would definitely give it a try. 

For what concerns the particle/arrays question, I have been wondering the same, (when I have done my java implementation of this code, I used "normal" 1 dimensional arrays) but i guess the reason lies in the architecture of jitter that basically uses matrices organised as pictures (n*m*planes, typically 4) as data repositories for all sort of data, and all the tools  are operating with those representations in mind. Plus this way you can easily do "shaders style" displacements using other pictures.

Finally, for what Gregory suggests, i.e. writing an external, I agree that would be the most elegant and sustainable way to do it, it should not be too complicated as it is mostly about porting an already existing java external to C, but still the amount of stuff I still don't understand about the way c and max relate, together with some serious pointers/memory handling allergy, make me shiver.. Plus I found a book for learning how to write msp externals, but nothing about Jitter... But if there are other people interested, why not make some sort of study group!  I'd be happy to! 

hey everyone, you might be interested in checking out 

 that covers some of these concepts (among others in the forums).

i spent a little (too much) time porting the threejs example linked in the first post to JS, which is my recommendation of what to try first. if you hit a cpu wall porting to C would be the next step. as mentioned in that thread, try and limit the number of global data structures you access in your loops (make local copies if necessary).

other strategies for these kinds of things involve data partitioning algorithms (b-trees, r-trees, 

point-connect.zip

hey everyone, you might be interested in checking out this thread that covers some of these concepts (among others in the forums). 

i spent a little (too much) time porting the threejs example linked in the first post to JS, which is my recommendation of what to try first. if you hit a cpu wall porting to C would be the next step. as mentioned in that thread, try and limit the number of global data structures you access in your loops (make local copies if necessary).  

other strategies for these kinds of things involve data partitioning algorithms (b-trees, r-trees, octrees, etc)

Thanks a lot for your message Rob, and for the patch!

I have done a somewhat similar code in Java this summer, following the information contained on that post you linked, making java calculate the connection array for the jit.gl.mesh. It performs way faster than js on big particle arrays (~4000 particles and such), but the bottleneck are really the methods to get jitter matrices in and out of java. In my case it was necessary to get the position of the particles by an analysis of images in real time, so I couldn't really stay all within java.

I can try cleaning the code asap and maybe publish it here. Still I am sure that if there would be a way to have access the matrix arrays in jit.gen, like you would do in java or js it would be so much easier! No plan of making it a feature at any point?

Thanks a lot for your message Rob, and for the patch!
I have done a somewhat  similar code in Java this summer, following the information contained on that post you linked, making java calculate the connection array for the jit.gl.mesh. It performs way faster than js on big particle arrays (~4000 particles and such), but the bottleneck are really the methods to get jitter matrices in and out of  java. In my case it was necessary to get the position of the particles by an analysis of images in real time, so I couldn't really stay all within java. 

Iterate in a jitter matrix and scripting in jitter.