Timo Rozendal

I have new puzzle: I am working on an external that loads audio files in buffer~s in a separate thread (actually: more than 1 at the same time). It runs fine... for a while (say, the first couple of thousand aiff files) and then my sysmem_newptr calls return 0, indicating that memory allocation has failed. This is confirmed by the debugger console message:

MaxMSP Runtime(31333,0x82829000) malloc: *** mmap(size=21168128) failed (error code=12)

*** set a breakpoint in malloc_error_break to debug

#3 0x900c3062 in large_and_huge_malloc ()

#9 0x16f6e87f in LoadBufferClass::readFile (this=0x4053d400) at /Users/Eboman/Desktop/my_work/MaxSDK-5.1.1/examples/bufferload/bufferload.cpp:162

#10 0x16f6efeb in LoadBufferClass::run (this=0x4053d400) at /Users/Eboman/Desktop/my_work/MaxSDK-5.1.1/examples/bufferload/bufferload.cpp:290

#11 0x16f6de1b in doRun (theLoadBufferClass=0x4053d400) at /Users/Eboman/Desktop/my_work/MaxSDK-5.1.1/examples/bufferload/bufferload.cpp:323

#12 0x0006040a in systhread_threadproc ()

However, as far as I know the total memory usage of max runtime fluctuates between 450 mb and 750 mb, but it is not steadily rising (and it is well below the application's memory limit). yes, I do allocate quite a bit of memory, but I free it properly as well (afaik)

is there a way to find more about the nature of this issue? or does any of you have clue as to where to start looking? (I tried some of the other debug tools xcode 3 has to offer, but no clue yet)

(and on a related note: how do I properly resize the sample memory of a buffer~, sysmem_ptrsize returns 0 and resizing it with t_freebytes results in this debug console message: "sysmem_freeptr: trying to free bad ptr 16bd4050" )

MaxMSP Runtime(31333,0x82829000) malloc: *** mmap(size=21168128) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

and this is the backtrace:
#0  0x9019a4b1 in malloc_error_break ()
#1  0x9019549f in szone_error ()
#2  0x900c2876 in allocate_pages ()
#3  0x900c3062 in large_and_huge_malloc ()
#4  0x900ba1c8 in szone_malloc ()
#5  0x900b9fb8 in malloc_zone_malloc ()
#6  0x900b9f4c in malloc ()
#7  0x0002be30 in sysmem_newptr ()
#8  0x11631c43 in sysmem_newptr ()
#9  0x16f6e87f in LoadBufferClass::readFile (this=0x4053d400) at /Users/Eboman/Desktop/my_work/MaxSDK-5.1.1/examples/bufferload/bufferload.cpp:162
#10 0x16f6efeb in LoadBufferClass::run (this=0x4053d400) at /Users/Eboman/Desktop/my_work/MaxSDK-5.1.1/examples/bufferload/bufferload.cpp:290
#11 0x16f6de1b in doRun (theLoadBufferClass=0x4053d400) at /Users/Eboman/Desktop/my_work/MaxSDK-5.1.1/examples/bufferload/bufferload.cpp:323
#12 0x0006040a in systhread_threadproc ()
#13 0x900ea055 in _pthread_start ()
#14 0x900e9f12 in thread_start ()

memory-allocation-issues

No clue about your memory allocation failure. But what I know for sure is that Max holds its own memory pool, and for this reason the debug tools in Xcode don't work.

About resizing a buffer~, can't you simply send it the "size" message?

hmm, didn't know about the max memory pool and that that 'blocks' xcode's tools. Is it somewhere documented how this works?

I tried the size message as an alternative, but it is not executing in line. I think it defers the call to the queue thread and for me it is important that all is happening in a non-standard max thread (so that it doesn't interfere with video processing).

I am afraid there is no documentation available for the max memory pool. I asked Joshua advice about how to have Leaks deal with it, and the answer was "you can't".

About the size message always being deferred, in fact it makes sense... it wouldn't surprise me if - besides efficiency concerns - there were thread-safety reasons behind this choice.

Having no idea of what you're trying to achieve, I wonder if using buffer~ is really the way to go. Maybe you should consider setting up a custom audio data container, adapted to the use you want to do with it - of course this would force you to re-invent the wheel by making your own timo.groove~, timo.record~ and so on...

resizing and thread safety is an issue yeah, so far the b_valid flag of buffer~ is working in my tests as long as I don't have waveform~ open where I continuously change the buffer contents.

I have a lead btw as to where the problem is, the memory allocation is not the problem in itself, but a symptom of another problem:

for every file I load I create a new thread. As soon as I have generated > 6000 threads since startup (there are only 10 active or so, the old ones are exited properly afaik) the memory allocation error pops up or the systems hangs and crashes. This is reproducible in a test external that I made that simple starts threads and stops them.

Maybe I should try an alternative to systhread_create to gain more insight into what's going on.

(shot in the dark: maybe the max memory pool reserves a certain amount of memory for every thread that is created?)

I have a lead btw as to where the problem is, the memory allocation is not the problem in itself, but a symptom of another problem:
for every file I load I create a new thread. As soon as I have generated > 6000 threads  since startup (there are only 10 active or so, the old ones are exited properly afaik) the memory allocation error pops up or the systems hangs and crashes. This is reproducible in a test external that I made that simple starts threads and stops them.
Maybe I should try an alternative to systhread_create to gain more insight into what's going on.

(shot in the dark: maybe the max memory pool reserves a certain amount of memory for every thread that is created?)


I have no experience of creating and destroying a large number of threads.

Are you sure there is no memory leak in the picture? you have a crude, but effective way of checking this out by watching the memory usage in the activity monitor.

But then I wonder - why, instead of continuously creating and destroying threads, don't you set up a large enough thread pool? This should be far more efficient!

Let me know if you find a solution - this topic really interests me.

The threads were not properly terminated. So apparently after not terminating >6000 threads weird things happen.

It appears calling systhread_exit(0) is not enough. You need to explicitly set the thread pointer to NULL in a max thread (using systhread_join).

However, systhread_join is stalling max so I call it just before my systhread_exit call by using a qelem.

The threads were not properly terminated. So apparently after not terminating >6000 threads weird things happen. 

However, systhread_join is stalling max so I call it just before my systhread_exit call by using a qelem. 

I confess you that I still wonder why you're not using a thread pool - if you say that you never use more than 10 threads at a time, 15 threads should be largely sufficient to ensure that your jobs are always immediately executed, and you would save the overhead of having to create and destroy lots of threads...

I'm not using a threadpool for simplicity, yeah maybe it is better to implement a proper queue which assigns a load command to a thread from the pool as soon as it comes available. Maybe I'll try that later.

Have you noticed the threadpool example in the SDK 5.1.7? that might be helpful...

I did have sdk 5.1.7 but it was not in there, but I redownloaded it and it's there (seems there are multiple versions of 5.1.7)

hey, thanks!
I did have sdk 5.1.7 but it was not in there, but I redownloaded it and it's there (seems there are multiple versions of 5.1.7)


memory allocation issues