Vincent Goudard

OK, starting a new thread for this problem which seemed somehow too unrelated from 

the thread I originally posted the message.

Test patch was the following : create a dozen of buffers and load them with some 300Mb soundfiles.

When reaching about 2 or 2.5Gb memory usage (as measured by OSX activity monitor), a message is kindly send to the max window to warn about memory limit : "buffer~: u809000206: out of memory reading my_3000mb_soundfile.wav"

If at this point, one starts to instantiate some graphical user interfaces (say a swatch for example), one eventually gets a crash.

Tests were made with Max5.1.9 and Max6.0.8 on the following systems :

- imac Intel Core i5 3.6 GHz - OSX 10.6.8 - RAM 8Go 1333 Mhz DDR3

- macbook pro 2.8Ghz Core 2 Duo - OSX 10.7.2 - 4Gb RAM

- macbook pro Core-i7 - OSX 10.8.2 - 8Gb RAM

- how to avoid the crash ? ( something like preventing a user from creating more interface and warning him/her with a message in max console would be utterly great — just like the buffer~ does)

- how to overcome this (pretty small) memory limitation ?

The example I gave with many buffers instanciation was an *example patch*, meant for C74 and anyone to reproduce the problem.

We have a real more complex patch in which we had trouble to find where the issue (crash) was, and please note that I am not exhausting the memory with an evil pleasure —viciously watchin the RAM-meter— in a non-debugging usual day :) ...

The buffers are instanciated automatically in a sound bank, and loading 2Gb of sound is something which happens pretty quickly when working on a whole music set. Now we have also some instruments (including GUI elements like swatch, which I took as an example in the test patch) that are also loadable dynamically (in poly~ object). Max5 also introduced a new patch format which is also much more memory-consuming than it was in Max4.x, and you can also easily reach more than 100mb of patchers.

We tried to avoid memory consumption by separating GUI from the audio/video processing algorithms, and followed some other 

 like the usefull ones Rob Ramirez mentionned in a recent post on the jitter list, but still face both low memory limit *and* the lack of low memory notification to prevent the crash.

One solution is to scan the max window to look for "out of memory" message kindly posted by buffer~ object, but one will not be warned from low memory caused by other objects.

I guess there should be ways to ask the system directly with the help of java/c object (shell ?), which would be helpful to warn the user that loading more sound/patch/whatever might be risky.

If anyone has ideas on how to circumvent crash in this context, I would be very glad to hear !

OK, starting a new thread for this problem which seemed somehow too unrelated from the thread I originally posted the message.

Test patch was the following : create a dozen of buffers and load them with some 300Mb soundfiles.
When reaching about 2 or 2.5Gb memory usage (as measured by OSX activity monitor), a message is kindly send to the max window to warn about memory limit : "buffer~: u809000206: out of memory reading my_3000mb_soundfile.wav"

If at this point, one starts to instantiate some graphical user interfaces (say a swatch for example), one eventually gets a crash.
(See crash report at the end)

Tests were made with Max5.1.9 and Max6.0.8 on the following systems :
- imac Intel Core i5 3.6 GHz - OSX 10.6.8 - RAM 8Go 1333 Mhz DDR3
- macbook pro 2.8Ghz Core 2 Duo - OSX 10.7.2 - 4Gb RAM
- macbook pro Core-i7 - OSX 10.8.2 - 8Gb RAM

The question are :
- how to avoid the crash ? ( something like preventing a user from creating more interface and warning him/her with a message in max console would be utterly great — just like the buffer~ does)
- how to overcome this (pretty small) memory limitation ?

Some more info on the context :
The example I gave with many buffers instanciation was an *example patch*, meant for C74 and anyone to reproduce the problem.
We have a real more complex patch in which we had trouble to find where the issue (crash) was, and please note that I am not exhausting the memory with an evil pleasure —viciously watchin the RAM-meter— in a non-debugging usual day :) ...

We tried to avoid memory consumption by separating GUI from the audio/video processing algorithms, and followed some other performance  enhancement rules like the usefull ones Rob Ramirez mentionned in a recent post on the jitter list, but still face both low memory limit *and* the lack of low memory notification to prevent the crash. 

One solution is to scan the max window to look for "out of memory" message kindly posted by buffer~ object, but one will not be warned from low memory caused by other objects.
I guess there should be ways to ask the system directly with the help of java/c object (shell ?), which would be helpful to warn the user that loading more sound/patch/whatever might be risky.

If anyone has ideas on how to circumvent crash in this context, I would be very glad to hear !


memory-limit-concerns-and-crash-was-allocating-more-memory-to-max

Hey - sorry if I came across a bit harsh in that other thread. Just a few wires crossed, as it turns out, now I see why you loaded up the available RAM (and Virtual Address Space).

I can only really speak to Windows here - but the symptoms you're experiencing seem right in line with what I'd expect to see from that workload in Max in 32bit Windows. Everything's just gotta be thrashing - from the HDDs to the D-Caches. If Max makes any heavy use of GDI+ (I don't know) - those GDI Objects are always ripe for depletion when graphical audiosoft enters the equation IME.

Streaming samples of that size into smaller buffers is one strategy you might employ. Basically, sidestep the OOM issue entirely.

Another, and I could be talking directly out of my rear at this point . . . do standalones run in a separate process and memory space? Spin some work out to standalones and use IPC of some sort so they can talk to each other. Everything gets its own largish chunk of VAS to play in. Inelegant as hell, but if it's possible - beats the machine falling over any day.

it was a good idea to start a different topic anyway, as the questions though related were somehow different.

We did consider splitting parts of the application in separate standalone, but this is not without difficulties, as one may expect... :)

It is essentially possible to communicate messages with OSC, audio with soundflower/jack (to some quite limited extent, tho), and video with syphon, but no shared GL render to my knowledge.

Splitting the app into separate standalone would also makes sense to prevent audio-related-crashes to trash a well-running-video-engine and vice versa. I had not thought of doing this for boosting the RAM limits though, but that's a good point !

We'll better wait for more memory space hopefully coming with 64bit version of Max.

The audio file we load are usually small bits but we load lots of them. It happens essentially at the opening of a set, so we can check for memory abuse at this step. But usually, the crash happens after that, at modules load time, with no warning from max, then.

It sounds quite ok to me if the GUI slow down, as long as you have time enough to save your set, or to unload some modules/sounds/video/...

I guess a soft like max with its [sexy as hell] "plug anything on anything" programming style is a good candidate to have the Crazy Talk with the machine indeed :)

We did consider splitting parts of the application in separate standalone, but this is not without difficulties, as one may expect... :)
It is essentially possible to communicate messages with OSC, audio with soundflower/jack (to some quite limited extent, tho), and video with syphon, but no shared GL render to my knowledge.
Splitting the app into separate standalone would also makes sense to prevent audio-related-crashes to trash a well-running-video-engine and vice versa. I had not thought of doing this for boosting the RAM limits though, but that's a good point !
We'll better wait for more memory space hopefully coming with 64bit version of Max.

The audio file we load are usually small bits but we load lots of them. It happens essentially at the opening of a set, so we can check for memory abuse at this step. But usually, the crash happens after that, at modules load time, with no warning from max, then.
It sounds quite ok to me if the GUI slow down, as long as you have time enough to save your set, or to unload some modules/sounds/video/... 

I guess a soft like max with its [sexy as hell] "plug anything on anything" programming style is a good candidate to have the Crazy Talk with the machine indeed :)


What you would really be doing in this context is exploiting the Virtual Address Space better.

Yes, you'd be making better use of your RAM - but the limits (for RAM, this is a pretty "hard" limit indeed!) still remain - the RAM you have is the RAM you have, obviously . . . and each process would still be limited to ~3GB of address space in a 32 bit system (again, Windows) with the 3GB switch thrown in BCDEDIT.

I'm being a little pedantic here, but it's a fairly important distinction depending on how you look at it.

Another thing to consider - even in 32bit - Windows can use more than 4GB of installed RAM under PAE. And 64bit OS can run 32bit applications . . . there are a few circumstances where you can actually remove some of the RAM limits, if not the virtual address space ones.

I thought OS X handled 32bit on 64bit thing fairly transparently due to differences in implementation from Windows - it'd be nice to have someone more versed in OS X to chime in (or Windows, I'm far from expert) here.

My turn to apologise - all this talk of Windows is a bit of a hijack here.

One application in 32 bit systems can take only up to 2GB for one process.

This info You can see in Procexp - alternative task manager 

http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

In the top of table - right click -> select columns -> Virtual size

And NOW You can see the moment of crashing Yours application if reached 2048K !!!

Hi to ALL !
One application in 32 bit systems can take only up to 2GB for one process.
This info You can see in Procexp - alternative task manager http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
In the top of table - right click -> select columns -> Virtual size
And NOW You can see the moment of crashing Yours application if reached 2048K !!!


You don't really know what you're talking about.

Memory limit concerns and crash [was : Allocating more memory to MAX?]