passing big matrices from max external to max

Alexis Baskind

Hi all,

I am struggling with a relatively simple problem: I created a max external in C that outputs among others a dictionary containing big float matrices (typically >50000x3). Those matrices are currently stored as three float arrays inside the dictionary, one for each column. When the size of the matrix remains "small" (tipycally less than 40000), everythings goes ok. But when the number of lines is too big, max crashes with the "Attempting excessively large memory allocation" in debug mode (Guard Malloc is on on Xcode).

I guess that the problem comes from the fact that a dictionary is not really made to handle such big amounts of data.

So my question: what's the best way to handle this, i.e. to pass big amounts of data to Max from the objects? My first idea was to create a Jitter matrix, but I did not find anywhere how to imbed a jitter matrix inside a dictionary in C. Is it possible? Does my external needs to be a MOP object for that? If someone could provide me some sample code, this would be very helpful.

Best,

Alexis

Timothy Place

Hi Alexis,

Without concrete code, I can only speculate. However, Jitter matrices are indeed the ideal way to pass large matrices efficiently in Max.

Without the background on what you wish to accomplish, I'm not sure why you wish to store this information in a dictionary. If the data of the matrix will remain within the confines of Max then you should have no problem passing matrices by reference (just passing the name of the matrix).

Cheers

Alexis Baskind

Hi Timothy,

Thanks for the answer. Here is the portion of code that aims at outputting the data:

    // dump extended calibration info for the accelerometer as a dictionary
    int i;
    t_atom output[3];
    t_atom accOffset[3];
    t_atom accScaling[3];
    
    t_atom *accCalDataCalSamples_x, *accCalDataCalSamples_y, *accCalDataCalSamples_z, *accCalDataNorm;
    
    dictionary_clear(x->AccCalInfoDict);
    
    dictionary_appendlong(x->AccCalInfoDict, gensym("numberOfSamples"), x->trackingData->accCalNumberOfRawSamples);
    
    // create the arrays of atoms that will be converted in an atom array when pushing to the dict
    accCalDataCalSamples_x = malloc(x->trackingData->accCalNumberOfRawSamples * sizeof(t_atom));
    accCalDataCalSamples_y = malloc(x->trackingData->accCalNumberOfRawSamples * sizeof(t_atom));
    accCalDataCalSamples_z = malloc(x->trackingData->accCalNumberOfRawSamples * sizeof(t_atom));
    accCalDataNorm = malloc(x->trackingData->accCalNumberOfRawSamples * sizeof(t_atom));
    for(i = 0; i<x->trackingData->accCalNumberOfRawSamples; i++) {
        atom_setfloat(accCalDataCalSamples_x + i, x->trackingData->accCalDataCalSamples[i][0]);
        atom_setfloat(accCalDataCalSamples_y + i, x->trackingData->accCalDataCalSamples[i][1]);
        atom_setfloat(accCalDataCalSamples_z + i, x->trackingData->accCalDataCalSamples[i][2]);
        
        atom_setfloat(accCalDataNorm + i, x->trackingData->accCalDataNorm[i]);
    }
    
    dictionary_appendatoms(x->AccCalInfoDict, gensym("calDataCalSamples_x"), x->trackingData->accCalNumberOfRawSamples, accCalDataCalSamples_x);
    dictionary_appendatoms(x->AccCalInfoDict, gensym("calDataCalSamples_y"), x->trackingData->accCalNumberOfRawSamples, accCalDataCalSamples_y);
    dictionary_appendatoms(x->AccCalInfoDict, gensym("calDataCalSamples_z"), x->trackingData->accCalNumberOfRawSamples, accCalDataCalSamples_z);
    
    dictionary_appendatoms(x->AccCalInfoDict, gensym("calDataNorm"), x->trackingData->accCalNumberOfRawSamples, accCalDataNorm);
    
    atom_setfloat(accOffset, x->accOffset[0]);
    atom_setfloat(accOffset+1, x->accOffset[1]);
    atom_setfloat(accOffset+2, x->accOffset[2]);
    dictionary_appendatoms(x->AccCalInfoDict, gensym("offset"), 3, accOffset);
    
    atom_setfloat(accScaling, x->accScaling[0]);
    atom_setfloat(accScaling+1, x->accScaling[1]);
    atom_setfloat(accScaling+2, x->accScaling[2]);
    dictionary_appendatoms(x->AccCalInfoDict, gensym("scaling"), 3, accScaling);
    
    atom_setsym(output, gensym("calData"));
    atom_setsym(output+1, _sym_dictionary);
    atom_setsym(output+2, x->AccCalInfoDictName);
    outlet_anything( x->x_status_outlet, gensym("calibrating_acc"), 3, output);
    
    free(accCalDataCalSamples_x);
    free(accCalDataCalSamples_y);
    free(accCalDataCalSamples_z);
    free(accCalDataNorm);

The four vectors accCalDataCalSamples_x, accCalDataCalSamples_y, accCalDataCalSamples_z and accCalDataNorm may be very big, as I wrote, more than 50000 samples (float numbers) is not unusual. The line that provokes the malloc breakpoint is the one with outlet_anything(). The number of bytes to be allocated, as reported by the debugger, is a huge number but a multiple of the number of samples, I guess that the problem comes from the fact that for each sample, a t_atom is being allocated, which takes too much memory.

My idea now is to create a jit_matrix with jit_object_new(), fill it with the samples, and (thanks to your advice), pass it to Max as a reference in the dictionary. Would something like the following line work?

dictionary_appendsym( x->AccCalInfoDict, gensym("CalData"), name_of_the_matrix);

Would it be possible (just for curiosity) to pass the Jitter matrix as an object? Something like:

dictionary_appendobject ( x->AccCalInfoDict, gensym("CalData"), (t_object ∗) my_jitterMatrix )

Best,

Alexis

Timothy Place

Hi Alexis,

The call to outlet_anything() having the crash could indicate a crash in any object or operation connected to the outlet of your object (or the outlets of their objects) so it doesn't illuminate the problem. But...

dictionary_appendsym( x->AccCalInfoDict, gensym("CalData"), name_of_the_matrix);

Should work well. Trying to use dictionary_appendobject()will not gain you anything, will be very difficult to debug, and ultimately only cause problems. So following your first intuition is recommended.

Good luck!