Uasmi Nasser

Did anyone successfully run a trained model in Max Msp (Node for Max)? Like running converted model (json + bin).

Most of the examples I found were about small networks built in tensorflow js, but not about using the trained models.

Currently I'm facing weird issues with creating Tensors (for example "unexpected stdin error" or I/O operation was killed) and I feel like it might be some Node for Max specifics, but I'm not familiar with it enough to debug it.

Would appreciate patch samples/scripts/links or anything else.

Did anyone successfully run a trained model in Max Msp (Node for Max)? Like running converted model (json + bin).
Most of the examples I found were about small networks built in tensorflow js, but not about using the trained models.
Currently I'm facing weird issues with creating Tensors (for example "unexpected stdin error" or I/O operation was killed) and I feel like it might be some Node for Max specifics, but I'm not familiar with it enough to debug it.
Would appreciate patch samples/scripts/links or anything else.

Hey,
Did anyone successfully run a trained model in Max Msp (Node for Max)? Like running converted model (json + bin).
Most of the examples 

max-msp-tensorflow-js

So are you trying to run Tensforflow from max node to create your model?

Nope, trying to run already trained model (inference)

I normally use pyCharm for machine learning development. But I like the idea of being able to use something like Tensorflow.js to load a model in Max and use it. This might help in configuring Tensorflow for node...

https://www.tensorflow.org/js/guide/nodejs

I normally use pyCharm for machine learning development. But I like the idea of being able to use something like Tensorflow.js to load a model in Max and use it. This might help in configuring Tensorflow for node...
https://www.tensorflow.org/js/guide/nodejs 

I'm actually close to finishing up the patch and script, I'll post the script here if I succeed .

Currently I see some drawbacks from Tensorflow.js CPU version, but I need to figure it out

Yeah checked it already, thx
I'm actually close to finishing up the patch and script, I'll post the script here if I succeed .
Currently I see some drawbacks from Tensorflow.js CPU version, but I need to figure it out

Hey Uasmi! Did you succeed? I am interested in learning how you did it/for what purpose!

Hey Uasmi! Did you succeed? I am interested in learning how you did it/for what purpose!

Let me know if you can share the patch!


I'm also curious and interested about this. Did you get it to work?

Unfortunately there are some drawbacks with tfjs that doesn't work in my particular case (specifically, if your input tensor has NCHW data format), but might work in yours. I also tried onnx js, but onnx can't convert my specific model (some layers are unsupported). Other than that it seems like it is possible, and I think I'll be able to help.

I'll gather the scripts and patch file, then I'll publish it to github and will post link here.

Unfortunately there are some drawbacks with tfjs that doesn't work in my particular case (specifically, if your input tensor has NCHW data format), but might work in yours. I also tried onnx js, but onnx can't convert my specific model (some layers are unsupported). Other than that it seems like it is possible, and I think I'll be able to help.
I'll gather the scripts and patch file, then I'll publish it to github and will post link here.


I've been using Heather Arthur's 'The Brain' succesfully to create neural networks inside MaxForLive (or just Max itself). It can be operated as it's downloaded using the require function in javascript.

Currently my goal is to classify drum samples (kick, clap, snare, CH, OH etc.). Last week I found an example to do non-realtime FFT using jit.fft. I compress the output to a 14 x 14 matrix and then even further to optain 53 cells of information per sample. After training a few hours the results look quite promissing. Most samples are analyzed into the right category with more then 90% accuracy.

XO by XLN is not just classifiying samples, but it also places them on a grid with x and y coordinates ranging from 0.0 to 1.0. That's what I want, but I don't know yet how I can get this kind of myself.

If wanted I can publish my patch here; there are no third-parties externals involved.

I've been using Heather Arthur's 'The Brain' succesfully to create neural networks inside MaxForLive (or just Max itself). It can be operated as it's downloaded using the require function in javascript. 

XO by XLN is not just classifiying samples, but it also places them on a grid with x and y coordinates ranging from 0.0 to 1.0. That's what I want, but I don't know yet how I can get this kind of myself.

If wanted I can publish my patch here; there are no third-parties externals involved.

@xanadu - that would be amazing if it’s not too much effort!

I will prepare my patch and then upload it. It will take some time to explain what’s going on.

AI-FFT.zip

Here's my current AI-FFT patch. It's an AMXD device for MaxForLive or Max itself. Please open it in the editor and keep the console visible.

The subdirectories Samples and Analysis are empty because of copyright issues. You can fill these with your own samples. The Sample directories are used for the testset. The Analysis directories are used for control tests.

It is controlled by the dark red message boxes at the top. The 'path' button is used for initialization. Button 5 can be pressed to doe a little test right away. To let this work for yourself you can start with button 2 to make a testset and then train it with buttons 3 and 4. More training leads to better results, Button 6 is for testing with sampes on the fly that were not part of the testset.

Please open ai-fft.js to look at the javascript code. The neural network itself is in the 'The-Brain.js' file.

To make a testset the samples are first scanned with a non-realtime jit.fft scanner. But 20 x 512 cells is way too much for the js code. So the fft results are compressed to 14 x 14 using fibonacce sequences. That way more info on the lower frequences and the start of the samples is put in the resulting set. But even 14 x 14 is too big. It is more compressed by just taking the length of the sample and row 1 and 6. For all columns and rows the standard deviation is added. It looks that the resulting sets can be distinguished quite well by the neural network.

This is a work in progress. Please tell us on this forum what you think of it.

The subdirectories Samples and Analysis are empty because of copyright issues. You can fill these with your own samples. The Sample directories are used for the testset. The Analysis directories are used for control tests. 

After publishing my patch for identification of the type of a drum sample I thought that gen, gen~, jit.jen or jit.gl.pix would be able to train the neural network on steroids. The code of Heather Arthur is very clear and I find a lot of video’s on Youtube on the inner workings of neural networks.

I will try to port the code or Heather Arthur to gen. Please give suggestions on what objects to use. should I use cpu or gpu and how can that be done?

I've tried to get a neural network working in GEN or JIT.GEN. No success. In GEN I can't use jit.matrices and there is an automatic timeout saying I'm probably running an endless loop. Another issue is in gen you have to program the whole neural network from the bottom up.

I've been able to install tensorflow.js in Node For Max., but no success on installing the C++ or GPU backend for higher speed. There are no unexpected errors. I'm taking my time learning Tensorflow.js.

The results are not good enough at the moment. After training a set of 104 samples for 5 minutes these where all recognized correctly , but from a testset of 104 different samples only 53% are recognized correctly.

The results are not good enough at the moment. After training  a set of 104 samples for 5 minutes these where all recognized  correctly , but from a testset of 104 different samples only 53%  are recognized correctly.

Also I think that training nets in Max is a problematic thing, I would rather train a neural net and then use a frozen model in Max, like exporting .pb as json+bin (through converter in tfjs) and then loading it

Did you try to import tfjs-node-gpu?
Also I think that training nets in Max is a problematic thing, I would rather train a neural net and then use a frozen model in Max, like exporting .pb as json+bin (through converter in tfjs) and then loading it


I have imported @tensorflow/tfjs-node and @tensorflow/tfjs-node-gpu. But on ‘require(...)’ both fail. I’m not sure if I’m missing some requirement or Node For Max doesn’t support machinecode addons at all.

actually traing a few thousand epochs with 140 samples with 225 cells each is quite fast. I‘va smashed a set of drum-samples together without much thought. I suspect the target types are not very consistent.

In Holland the lockdown Is being slightly released, I can go to the Zoo next week! But I will still have ample time to figure out neural networks in MaxForLive.

In Holland the lockdown  Is being slightly released, I can go to the Zoo next week! But I will still have ample time to figure out neural networks in MaxForLive.

@XANADU What exactly is the error you are seeing? There is no general restriction on loading native extensions within Node For Max so I'd assume it's a more specific error / issue you are seeing. If you provide a more detailed error report / log output and maybe even share a stripped down project with steps to reproduce I'm happy to help.

The problem with ‘tfjs-node’ and ‘tfjs-node-gpu’ is published on several fora/forums on internet. Perhaps ‘gyp’ is not installed properly or python.exe is not on the path. This is not a critical problem. My model can recognize 12 types of drumhits with 87% being correct. The other 13% are Kicks being recognized as Toms and vice versa. Their also is bleeding between Closed hats, Open hats, Snares and Cymbals. This behavior was expected. In fact this score is better then XO on the same samples. Optimal traing is reached within half an hour and only has to be done once.

A more severe problem was that saving to disk and reloading a model does not work. This also is a known problem without a good solution. I’ve got the function getWeights() working and I think setWeights() is working to now. I still have to check if this workaround really does the job.

I will clean up my Max-device and then publish it on this forum or on MaxForLive. I will document these issues. Perhaps it will inspire others and I’m hoping for tips and improvements. All my devices are free and will remain free for at least another year.

The problem with ‘tfjs-node’ and ‘tfjs-node-gpu’ is published on several fora/forums on internet. Perhaps ‘gyp’ is not installed properly or python.exe is not on the path. This is not a critical problem. My model can recognize 12 types of drumhits with 87% being correct. The other 13% are Kicks being recognized as Toms and vice versa.  Their also is bleeding between Closed hats, Open hats, Snares and Cymbals. This behavior was expected. In fact this score is better then XO on the same samples. Optimal traing is reached within half an hour and only has to be done once.

Max MSP + Tensorflow.js