TensorFlow.js in Max/MSP with a practical example

Greetings, this topic is a translation of a tutorial I wrote for my blog (in Italian) Musica Profonda, where I talk about music and deep learning. It is an example of how TensorFlow can be used inside Max, using it and the freely available model YAMNet to recognize sounds featured in a file.
First of all, you can download the full patch here (you also find it attached to this post). The only external library you will need is HISSTools (available directly from Max Package Manager), used to easily downsample the files you are loading to 16 kHz (which is the sample rate used by the neural network).
First of all, let's create a buffer that we will populate with the sound file we are going to use for timbre recognition:

Then, let's create an object node.script where we put this code (functions are seen in detail soon):

const Max = require('max-api');
const tf = require('@tensorflow/tfjs');
const fs = require('fs');
const modelUrl = 'https://tfhub.dev/google/tfjs-model/yamnet/tfjs/1';
var channels = 1;

buffer = [];
main_sounds = [];
let model;

async function init()
{
    model = await tf.loadGraphModel(modelUrl, { fromTFHub: true });
}

async function readbuf_async(file) 
{
    data = fs.readFileSync(file);
    buffer = [];
 

    for (o = 0; o < data.length - (4 * channels); o += 4 * channels)
    {
        buffer.push(data.readFloatBE(o));
    }
 

    await Max.post("Buffer filled!");
 };

async function predict()
{    
    waveform = tf.tensor(buffer);
    const [scores, embeddings, spectrogram] = model.predict(waveform);
      maxscore = scores.mean(axis=0).argMax();
        
        scores_length = scores.shape[0];
     main_sounds = [];
     
        arr = scores.arraySync();
     for (i = 0; i < scores_length; i++)
    {
        main_sounds[i] = arr[i].indexOf(Math.max.apply(Math, arr[i]));
    }
     
        await Max.post("Prediction done!");
    await Max.outlet(main_sounds);
}    

Max.addHandler('init', () => 
{
    init();
});

Max.addHandler('clearbuf', (sample) => 
{
    buffer = [];
});

Max.addHandler('setchans', (chans) => 
{
    channels = chans;
});

Max.addHandler('readbuf', (file) => 
{
    readbuf_async(file);
});

Max.addHandler('predict', () =>
{
        predict();
});

We attach these messages to the node.script object:

• script npm install @tensorflow/tfjs: to install TensorFlow.js

• script start: to start the script, a script stop message can be useful, too

• init: to call the function with the same name that loads the YAMNet model

• predict: to call the function with the same name that make the prediction

• setchans $1: to tell the script the number of channels of the buffer, with $1 linked to the last outlet of the info~ object. Keep in mind that for the actual prediction only the first channel will be used

• readbuf $1: to tell the script the path of the temporary file which the buffer has been written to. The $1 argument is received by a JavaScript (inside a js object, the sendbuf.js you can see in the first picture) that automatically recovers the file path:

autowatch = 1;
inlets = 1;
outlets = 1;

function bang()
{
    var patcher_dir = this.patcher.filepath.replace(patcher.name + ".maxpat", "");
    outlet(0, patcher_dir + "samplebuf.tmp");
}

The same script sends the filename and its path also to the message samptype float32, writeraw $1, to be linked to the buffer, that will be read by Node.js in the readbuf_async function.

Let's go back to the Node script: this has an outlet which sends to a multislider the timbre categories predicted by the function predict: it is a series of numbers from 1 to 521, each one corresponding to most appropriate category to classify a chunk of 480 milliseconds of the buffer content.
Playing the buffer with play~ and incrementing a counter every 480 ms, we will be able to know which is the current most represented sound. A coll links each category index to its name, so that it can displayed in a comment:

This is just a primer and it can be surely made in more efficient ways, but I wanted to share with you some of my discovering as a beginner in this area.
Let me know if it has been useful, and what are your experiences with Max and neural networks!