Mathieu Chamagne

In some situation, AFAIK there is no way to avoid creating lots of new 'big' symbol very often in Max, and it seems that it can eat a lot of RAM very quickly...

(filling *I guess* the Max symbol hash table)

---> please open patch below (everything is explained in it) :

If I understand things correctly, as long as data stay in dictionaries, no new symbol is created.

but when you want to send a dict over network, the only way I know is using dict.serialize, and then the whole content of the dict is turned into a symbol.

a way to turn dict in some kind of 'FullPacket' message to send directly to udpsend ?

(and in the other way fullPacket coming from udpreceive --> dict)

any good idea on a way to avoid this problem ?

In some situation, AFAIK there is no way to avoid creating lots of new 'big' symbol very often in Max, and it seems that it can eat a lot of RAM very quickly...
(filling *I guess* the Max symbol hash table)

This is probably not a bug report...
but maybe more a feature request ?

If I understand things correctly, as long as data stay in dictionaries, no new symbol is created.
but when you want to send a dict over network, the only way I know is using dict.serialize, and then the whole content of the dict is turned into a symbol.
Maybe there's something missing, here ?
a way to turn dict in some kind of 'FullPacket' message to send directly to udpsend ?
(and in the other way fullPacket coming from udpreceive --> dict)

________________
Mathieu Chamagne
http://mathieuchamagne.com
http://www.maxobjects.com


how-to-avoid-flooding-max-symbol-hash-table

Maybe make your own json parser in javascript, and extract the data without having to stringify it in max first...?

well... I need to receive json via udp.. so as soon as udpreceive outputs the 'stringified' json, it will creates a new symbol before I send it to a js object !

but maybe doing everything in java could be a solution (receive udp + json parsing to dict) ?...

but maybe doing everything in java could be a solution (receive udp + json parsing to dict) ?...
...a good opportunity to learn Java :-s

To avoid to flood the symbol table with NTP timestamps i split the 64-bit value in 8 words (coded with HEX).

in the example I sent, I only change the 'timestamp' key of the dict before sending it to dict.serialize ;

but in real life, the whole content of the dictionary changes @ each frame (even it's structure), so I can't see a way to turn it all into a list of ints :-s

in the example I sent, I only change the 'timestamp' key of the dict before sending it to dict.serialize ;
but in real life, the whole content of the dictionary changes @ each frame (even it's structure), so I can't see a way to turn it all into a list of ints :-s

https://cycling74.com/forums/dict-in-java/

(and writing my own external is unfortunately not within my competence...)

ups.. no dict support in Java yet : https://cycling74.com/forums/dict-in-java/
I need to find another solution...
(and writing my own external is unfortunately not within my competence...)


instead of using udpreceive, which interprets and outputs udp data as ascii and then converts to strings (which is very limiting, they should have an option to change that), you could use something else which spits out the raw bytes. there is an external called sadam.udpReceiver that does this, so max simply sees it as a list of integers-- voila! no symbol table intervention. then convert to ascii, parse and process in javascript, so your js object just outputs the data into the max environment you are interested in in list form.

Thanks Mr Floating Point', that sounds like a good idea :-)

(I guess the same result could also be done with udpreceive in CNMAT mode (outputting fullPacket message) + aka.datagram)

I just hope I won't reach any limitation with list length, or add too much cpu overhead doing all these jobs in a js @100 fps ...

I'll of course let you know if I find a working solution.

Thanks Mr Floating Point', that sounds like a good idea :-)
I'll try using sadam.udpReceiver + js
(I guess the same result could also be done with udpreceive in CNMAT mode (outputting fullPacket message) + aka.datagram)

I think there is about a 2 ms latency and another 2ms jitter with sadam.udpReceiver so theoretically you could receive up to 200-250 hz packet rate. The size of the packet doesn't seem to affect the UDP transmission rate, in my experience up to a ceiling of about 24000 bytes (although theoretically it should go to about 65000)

I notice your example packet is a little less than 6000 bytes. So it should be fine. But be careful with processing lists of this size in max, as objects such as atoi only go up to 4096 (and zl objects default to 256, but you know that).

Let me know you you travel, I'm interested.

I notice your example packet is a little less than 6000 bytes. So it should be fine. But be careful with processing lists of this size in max, as objects such as atoi only go up to 4096 (and zl objects default to 256, but you know that).
Let me know you you travel, I'm interested.


just found out thresh has a limit of 4096 too btw

I did also noticed that Max atoi & itoa are limited to 4096 elements...

so your advice seems to be the solution :

I receive the packet with sadam.udpReceiver, so I get a big array in Max (that fortunately do not eats ram !)

But for now, I'm looking for an efficient way of making the itoa conversion inside js :

should I really iterate on each element with string.fromCharCode() function ?

couldn't it be done on the whole array in 'one shot' ?

in this example, an array of ints outputs nothing...

function anything()
{
    var a = arrayfromargs(arguments);
    outlet(0, String.fromCharCode(a));
}

so your advice seems to be the solution :
I receive the packet with sadam.udpReceiver, so I get a big array in Max (that fortunately do not eats ram !)

But for now, I'm looking for an efficient way of making the itoa conversion inside js :
should I really iterate on each element with string.fromCharCode() function ?
couldn't it be done on the whole array in 'one shot' ?
(yes, I'm lousy at js !)

in this example, an array of ints outputs nothing...
but it works fine with one single int.
what's wrong in this code ?

function anything()
{
	var a = arrayfromargs(arguments);
	outlet(0, String.fromCharCode(a));
}

Of course it will work only if in the parsed OBJECT the pros i extract actually exists.

I mean all the ids... those could be any property of any object inside the parsed structure.

    var myascii=[];for(myf=0;myf','?','@'];
    var myextrac2=['[','\\',']','^','_',''];
    var myextrac3=['{','|','}','~',''];
    for(myf=0;myf

In a short LEAPFORMAX website will be updated and all memory leak s will finally be a thing of the past.

This should be what you are looking for.
Of course it will work only if in the parsed OBJECT the pros i extract actually exists.
I mean all the ids... those could be any property of any object inside the parsed structure.

	var myascii=[];for(myf=0;myf','?','@'];
	var myextrac2=['[','\\',']','^','_',''];
	var myextrac3=['{','|','}','~',''];
    for(myf=0;myf

In a short LEAPFORMAX website will be updated and all memory leak s will finally be a thing of the past.


This patch will receive some JS object via an underlying sadam.udpreceiver.

decode, JSON.parse, and properties extraction only when asked.

There is a very handy message supported that is "props".

the output will grab the props from the parsed object.

for example if we send {A:'0',B:{C:'99'}}

This is very handy to support virtually any kind of prop from any kind of object structure.

NOTE that sadam.udpreceiver is necessary in this version

NOTE that in this preliminary version if a prop does not exist in the object, the entire output will fail (TODO:try cath every prop)

NOTE that you can ignore tcpigolo-offline.js it's written for nodeJS :in this context you can see it as just some program sending some JSON via udp.

OK this is a more functional preview.
This patch will receive some JS object via an underlying sadam.udpreceiver.
decode, JSON.parse, and properties extraction only when asked.

There is a very handy message supported that is "props".
the output will grab the props from the parsed object.
for example if we send {A:'0',B:{C:'99'}}
and props be props A B.C
the output will be
0 99

NOTE that sadam.udpreceiver is necessary in this version
NOTE that in this preliminary version if a prop does not exist in the object, the entire output will fail (TODO:try cath every prop)
NOTE that you can ignore tcpigolo-offline.js it's written for nodeJS :in this context you can see it as just some program sending some JSON via udp.


AFAIK EVERY occurring string istance is hashed and stored as symbol forever until max quits.

to check if this is your problem have a NORMAL MESSAGE in max with this exact text inside (respect spaces)

When you click on it, the MAX WINDOW wil tell you the total number of symbols. You have a problem if it increases constantly every second.

It's got to be the same old reason.
AFAIK EVERY occurring string istance is hashed and stored as symbol forever until max quits.
to check if this is your problem have a NORMAL MESSAGE in max with this exact text inside (respect spaces)
------------------
; max size
------------------
When you click on it, the MAX WINDOW wil tell you the total number of symbols. You have a problem if it increases constantly every second.

Check out our solution on https://www.leapformax.com


Also posted the final statements of this thread here.

https://sites.google.com/a/leapformax.com/tcpigoleap/home/what-was-the-memory-issue

Also posted the final statements of this thread here.https://sites.google.com/a/leapformax.com/tcpigoleap/home/what-was-the-memory-issue


To avoid to flood the symbol table with NTP 

timestamps i split the 64-bit value in 8 words (coded with HEX).

That’s also a workaround for the 32-bit limit of integers.

To avoid to flood the symbol table with NTP cat mariotimestamps i split the 64-bit value in 8 words (coded with HEX).

That’s also a workaround for the 32-bit limit of integers.


how to avoid flooding Max symbol hash table ?