Forums > Dev

resizing a buffer~ in an external

January 19, 2008 | 9:19 pm

hi, i’m trying to write to a buffer in an external i’m making, and this involves resizing it. i’m wondering what the proper way to do this is.

i’ve tried something like:

free(b->b_samples);
b->b_samples = (float*)malloc(size*sizeof(float));

and updating the buffers size feilds etc, then just writing to the new b2->b_samples… but i think this is all very wrong. i get memory violation errors when i try to free the old b_samples for starters.

i’d appreciate any help


January 19, 2008 | 9:29 pm

pete wrote:
> hi, i’m trying to write to a buffer in an external i’m making, and this involves resizing it. i’m wondering what the proper way to do this is.
>
> i’ve tried something like:
>
> free(b->b_samples);
> b->b_samples = (float*)malloc(size*sizeof(float));

You have to use the same memory de-/allocation routines that were used
to allocate the memory. That’s most likely t_getbytes() and
t_freebytes() but I don’t know for sure.
Additionally, if you change the size you have to set certain values to
correspond with the new size. Here’s a code snippet that shows what I’m
doing. It seems to work…:

———-

x->x_buffer->b_size = size; // total size in samples (all channels)
x->x_buffer->b_nchans = x->x_channels;
x->x_buffer->b_frames = x->x_buffer->b_size / x->x_buffer->b_nchans;
x->x_buffer->b_outputbytes = sizeof(float);

———-

Olaf


January 19, 2008 | 10:59 pm

cheers for that, it’ll probably work when i can work out what library i need to include for t_getbytes and t_freebytes.. any ideas?


January 19, 2008 | 11:59 pm

pete wrote:
> cheers for that, it’ll probably work when i can work out what library i need to include for t_getbytes and t_freebytes.. any ideas?

Same as when writing DSP externals: library is MaxAudio.lib and include
z_dsp.h.
Important: Make sure to give your external a name with a ~ so that Max
knows it’s using DSP functions (even if it’s just only the memory
allocation) otherwise your externals would crash in Max Runtime (at
least that’s what used to happen with Max 4.5 on OS X).

BTW, there is also t_resizebytes() that you can use.

Olaf


January 20, 2008 | 12:29 am

thanks! it works now. i still get access violation something on the first t_freebytes call, but only in debug mode so i can live with that.

i couldnt find documentation anywhere of t_freebytes/getbytes/resizebytes though, which is a bit odd


January 20, 2008 | 12:32 am

i thought originally it would just be a max external rather than msp, because it has no perform function. the "offline" use of buffer~s is the only msp type aspect of it. should i still give it a ~ (twiddle)?


January 20, 2008 | 8:17 am


January 20, 2008 | 7:59 pm

t_freebytes() < --wrapper for free()
getbytes() < -- wrapper for calloc()
resizebytes() < -- wrapper for realloc()

On Jan 19, 2008 4:29 PM, pete

wrote:
>
> thanks! it works now. i still get access violation something on the first t_freebytes call, but only in debug mode so i can live with that.
>
> i couldnt find documentation anywhere of t_freebytes/getbytes/resizebytes though, which is a bit odd
>
>


January 20, 2008 | 9:35 pm

> hi, i’m trying to write to a buffer in an external i’m making, and this involves resizing it.

You may find this thread relevant:

http://tinyurl.com/yomu5y

Eric


January 20, 2008 | 9:48 pm

Quote: bbnickell wrote on Sun, 20 January 2008 20:59
—————————————————-
> t_freebytes() < --wrapper for free()
> getbytes() < -- wrapper for calloc()
> resizebytes() < -- wrapper for realloc()
—————————————————-

Not so. getbytes() & Co. look a bit like malloc() & Co., or even like NewPtr() & Co. (the One True Memory Allocation Methods). But they are very, very different. The one is not simply a wrapper for the other. If you mix up the different families of memory allocation methods, you will shoot yourself in your most sensitive anatomy.

To the original query: the getbytes() routines are documented in the SDK WritingExternals.pdf (where else?). In the version that comes with the 4.5.5 SDK, look at pp. 82, 101-102.


January 20, 2008 | 10:21 pm

Quote: Olaf Matthes wrote on Sun, 20 January 2008 00:59
—————————————————-
> Important: Make sure to give your external a name with a ~ so that Max
> knows it’s using DSP functions (even if it’s just only the memory
> allocation) otherwise your externals would crash in Max Runtime (at
> least that’s what used to happen with Max 4.5 on OS X).
—————————————————-

Olaf… are you sure about that? It sounds more like urban legend, to be honest.

In order to use DSP methods, your object needs to call dsp_initclass() inside main(), and its first component needs to be a t_pxobject instead of a t_object. The tilde is simply a naming convention, no more. AFAICT it was primarily instigated to provide a simple way to distinguish between + and +~ (and analogously for about two thousand other object pairs).

I do recall that at least one C74 quasi-dsp object went through most of its beta incarnation without the tilde, which is why I find the statement that a tilde-less object name would crash the RT a little surprising.


January 20, 2008 | 10:55 pm

Peter Castine wrote:
> Quote: Olaf Matthes wrote on Sun, 20 January 2008 00:59
> —————————————————-
>> Important: Make sure to give your external a name with a ~ so that Max
>> knows it’s using DSP functions (even if it’s just only the memory
>> allocation) otherwise your externals would crash in Max Runtime (at
>> least that’s what used to happen with Max 4.5 on OS X).
> —————————————————-
>
> Olaf… are you sure about that? It sounds more like urban legend, to be honest.
>
> In order to use DSP methods, your object needs to call dsp_initclass() inside main(), and its first component needs to be a t_pxobject instead of a t_object. The tilde is simply a naming convention, no more. AFAICT it was primarily instigated to provide a simple way to distinguish between + and +~ (and analogously for about two thousand other object pairs).

Sorry, it was late at night… when I look at my code now (late at night
again) I can tell that I used a name with a ~, dsp_initclass() and that
there is a t_pxobject instead of t_object. So it’s basically a DSP
object without perform routine.

> I do recall that at least one C74 quasi-dsp object went through most of its beta incarnation without the tilde, which is why I find the statement that a tilde-less object name would crash the RT a little surprising.

Yes, it was crashing in Runtime when I was calling t_getbytes() and
friends in a non-DSP external. This was with Max 4.5.? on PPC MacMinis.
I will meet these machines again in February and can do some more tests.
It never crashed on my Windows machine (in RT), but on those Macs my
patch didn’t work in Runtime but did work in regular version. After
making by external a ‘pseudo-DSP’ object it stopped crashing.

Olaf


January 21, 2008 | 5:20 am

The point of the thread I mentioned above is that you really don’t need to go through all of this hassle. It’s much easier to just send a "size" message to your buffer inside your code, and let Max take care of memory management for you. Although I’m curious if anyone can think of a reason why it would be better to take on memory management for yourself.

Eric


January 21, 2008 | 9:34 pm

Quote: Eric Lyon wrote on Mon, 21 January 2008 06:20
—————————————————-
> The point of the thread I mentioned above is that you really don’t need to go through all of this hassle. It’s much easier to just send a "size" message to your buffer inside your code, and let Max take care of memory management for you. Although I’m curious if anyone can think of a reason why it would be better to take on memory management for yourself.
—————————————————-

Good point.

My responses were intended to clarify some confusing statements made in the thread, and people really really ought to know *how* to manage memory correctly in Max.

But, of course, the most direct solution to the OPQ is to sit down with the SDK documentation and read how to use pass messages to Max objects from the C API. And while there, reviewing Max memory management isn’t a bad idea, either!


January 22, 2008 | 9:24 am

void *getbytes(short size)
void* freebytes(void *ptr, short size)

The Writing Externals SDK says that this set of memory allocation
routines are to be used only for interrupt-level memory allocation.
These are supposed to be reentrant functions capped off with 32767
bytes entered as short size in which anything bigger than 16384 bytes
passes allocation off to NewPtr(). NewPtr() is not a reentrant memory
allocation function and will complain if called at the interrupt level
by returning size 0. NewPtr() is also specific to Max OSX memory
allocation so what gets used in the case of interrupt level memory
allocation greater than 16384 bytes in Win32…something which is also
like NewPtr or a wrapper to something else? I’ve never seen this
brought up and I believe you even asked what the fundamental
differences are between malloc() and NewPtr() on native-OS memory
allocation routines:

http://www.synthesisters.com/hypermail/max-msp/Jan06/37517.html

The sysmem_* allocation routines outside of interrupt memory
allocation all have their descriptions more or less in the docs as
"Like malloc()…" or "(insert native Mac OS allocation routine)" –
unless I’m completely missing something I haven’t seen anything ever
discussed as to how these sysmem calls are more portable or more safe
than malloc(), calloc(), realloc() and free(). If anyone can offer a
technical explanation I would be very much appreciate it as the Memory
Management section of the SDK doesn’t really offer a deeper
explanation into this particular matter.

Miller had it down with Pure Data–the functions below had everything
wrapped around portable malloc routines. The ISO/IEC 9899:1999 ANSI C
standard ("C API" is a misnomer) says "the functions in the standard
library are not guaranteed to be reentrant and may modify objects with
static storage duration" so it’s left to the compiler implementation
whether or not to implement C library functions as reentrant or not.
Perhaps it’s this reason that the traditional getbytes/freebytes
functions differ in their parameters with the Max API by limiting it
to a set of reentrant routines with a short type. Pure Data
getbytes()/freebytes():

/* Copyright (c) 1997-1999 Miller Puckette.
* For information on usage and redistribution, and for a DISCLAIMER OF ALL
* WARRANTIES, see the file, "LICENSE.txt," in this distribution. */

void *getbytes(size_t nbytes)
{
void *ret;
if (nbytes < 1) nbytes = 1;
ret = (void *)calloc(nbytes, 1);
#ifdef LOUD
fprintf(stderr, "new %lx %dn", (int)ret, nbytes);
#endif /* LOUD */
#ifdef DEBUGMEM
totalmem += nbytes;
#endif
if (!ret)
post("pd: getbytes() failed — out of memory");
return (ret);
}

void *resizebytes(void *old, size_t oldsize, size_t newsize)
{
void *ret;
if (newsize < 1) newsize = 1;
if (oldsize < 1) oldsize = 1;
ret = (void *)realloc((char *)old, newsize);
if (newsize > oldsize && ret)
memset(((char *)ret) + oldsize, 0, newsize – oldsize);
#ifdef LOUD
fprintf(stderr, "resize %lx %d –> %lx %dn", (int)old, oldsize,
(int)ret, newsize);
#endif /* LOUD */
#ifdef DEBUGMEM
totalmem += (newsize – oldsize);
#endif
if (!ret)
post("pd: resizebytes() failed — out of memory");
return (ret);
}

void freebytes(void *fatso, size_t nbytes)
{
if (nbytes == 0)
nbytes = 1;
#ifdef LOUD
fprintf(stderr, "free %lx %dn", (int)fatso, nbytes);
#endif /* LOUD */
#ifdef DEBUGMEM
totalmem -= nbytes;
#endif
free(fatso);
}

On Jan 21, 2008 1:34 PM, Peter Castine

wrote:
>
> Quote: Eric Lyon wrote on Mon, 21 January 2008 06:20
> —————————————————-
> > The point of the thread I mentioned above is that you really don’t need to go through all of this hassle. It’s much easier to just send a "size" message to your buffer inside your code, and let Max take care of memory management for you. Although I’m curious if anyone can think of a reason why it would be better to take on memory management for yourself.
> —————————————————-
>
> Good point.
>
> My responses were intended to clarify some confusing statements made in the thread, and people really really ought to know *how* to manage memory correctly in Max.
>
> But, of course, the most direct solution to the OPQ is to sit down with the SDK documentation and read how to use pass messages to Max objects from the C API. And while there, reviewing Max memory management isn’t a bad idea, either!
> –
> Peter Castine
> –
> Peter is in dire need of a new Facebook tagline. Not to mention a new .sig
>
>


January 22, 2008 | 12:25 pm

back to the original question in case anyone is interested, i’m sending buffer~ a size message now like eric suggested and all is good. i’m used to higher level OO, so it does seem kind of wrong to me to be doing memory management across objects.

thanks for all the help


January 22, 2008 | 12:52 pm

> In order to use DSP methods, your object needs to call dsp_initclass() inside main(), and its first component needs to be a t_pxobject instead of a t_object. The tilde is simply a naming convention, no more. AFAICT it was primarily instigated to provide a simple way to distinguish between + and +~ (and analogously for about two thousand other object pairs).

i tried changing my object’s first component to t_pxobject and adding a call to dsp_initclass() in main(), because i gathered this is the right thing to do as my object uses buffer~ and should therefore have a ~ itself. but doing this made max crash whenever i turned dsp off. also, i noticed that buffer~ itself does not have a t_pxobject as its first component.

so, what i’m trying to say is that my object does not have a perform routine, doesnt call dsp_initclass() and does not have a t_pxobject, so should it realy have a ~? not that it matters much, i’m just curious about conventions.


January 22, 2008 | 2:27 pm

Quote: bbnickell wrote on Tue, 22 January 2008 10:24
—————————————————-
> void *getbytes(short size)
> void* freebytes(void *ptr, short size)
>
> The Writing Externals SDK says that this set of memory allocation
> routines are to be used only for interrupt-level memory allocation.
—————————————————-

I don’t see the word "only" in the documentation. And lots of existing code uses the getbytes() family exclusively, regardless of interrupt concerns.

FTR, I wrote a couple of externals that tested for isr(), used native memory allocation if safe and getbytes() otherwise, stored a flag to note which kind of allocation was used so that the proper deallocation would be called, and even deferred deallocation in case that would be necessary.

All of which was pretty much a waste of effort on my part.

My rule of thumb: Use native memory allocation for big stuff. Use getbytes() for small stuff. Prends-le ou laisse.

Note that Miller wrote in his 1980s papers on developing Max that he developed getbytes() & Co. to be particularly efficient. And compared to the contemporaneous malloc() implementations, he succeeded. I’ll have to dig up the ref in case anyone cares.

It’s all well and good to say what Pd does things much better than Max/MSP, but Pd does not have to be compatible with 20 years of Max patches & externals. I cannot say whether this is an issue in this particular case, but it is worth taking note of.

Regarding cross-platform issues. Getbytes() has the advantage that you don’t have to worry about host OS. Alternatives for native memory allocation are:

- sysmem_newptr() & Co.: wrapper functions to whatever the host malloc() is. Use like malloc() & co. Only drawback: overhead for an extra procedure call (setting up stack etc.) Advantage: you don’t have to worry about the host OS.

- Conditionally compile

#if WIN_VERSION
#define NewPtr(x) malloc(x)
// etc. etc. etc.
#endif

Quick, and only requires a few extra lines of code. You could also #if… #define in the other direction. All a question of taste. Since OS X this isn’t absolutely necessary, because you can simply malloc() on either Mac or Windows. You can even malloc() in interrupt nowadays, it’s just that malloc() may take more time than you want to wait in the middle of, say, an MSP perform() routine.

Hope this helps,
P.


January 22, 2008 | 2:39 pm

Quote: peterworth@gmail.com wrote on Tue, 22 January 2008 13:52
—————————————————-

> i tried changing my object’s first component to t_pxobject and adding a call to dsp_initclass() in main(), because i gathered this is the right thing to do as my object uses buffer~ and should therefore have a ~ itself. but doing this made max crash whenever i turned dsp off. also, i noticed that buffer~ itself does not have a t_pxobject as its first component.
>
> so, what i’m trying to say is that my object does not have a perform routine, doesnt call dsp_initclass() and does not have a t_pxobject, so should it realy have a ~? not that it matters much, i’m just curious about conventions.
—————————————————-

I think buffer~ & co. answer your second paragraph.

For the first paragraph: hard to say without seeing code.

Again: the tilde is a convention. AFAICT, the Max engine does *not* look at the object name, strchr() on ‘~’ and then decide whether or not to do striped patch cords. It is the call to dsp_initclass() that does the magic.


January 31, 2008 | 10:35 am

I’ve been reading through the Max external developers manual trying to understand whether or not it’s possible to allocate more memory in Max using new (in C++) while audio processing is running, but I’m having a fair amount of difficulty understanding the manual despite reading through it several times.

The manual talks about running at interrupt level for instance, and apparently, you enter interrupt level when you put Max in overdrive. Do you also enter interrupt level when you turn on audio processing in MSP? And what exactly is interrupt level? My guess would be that each external method is then called from a critical section so that the call can’t be interrupted. But if this is the case why shouldn’t you be able to allocate more memory? If the problem with allocating more memory is that the memory manager isn’t reentrant, then you shouldn’t have any problems allocating more memory from a critical section. Maybe, entering interrupt level means that an external can be interrupted at any point by another thread, but if that’s the case, then it should be no problem to allocate more memory using new as long as you call it from a critical section.

Hm… Could someone give me some clarification about interrupt level please?

Best,
Greg


February 1, 2008 | 12:55 pm

On 2008 Jan 31, at 4:35 AM, Greg Kellum wrote:

> I’ve been reading through the Max external developers manual trying
> to understand whether or not it’s possible to allocate more memory
> in Max using new (in C++) while audio processing is running, but I’m
> having a fair amount of difficulty understanding the manual despite
> reading through it several times.
>
> The manual talks about running at interrupt level for instance, and
> apparently, you enter interrupt level when you put Max in overdrive.
> Do you also enter interrupt level when you turn on audio processing
> in MSP? And what exactly is interrupt level?

Hi,

Basically, you want to avoid doing expensive operations (like memory
allocation and deallocation) in anything called directly by the
scheduler or the in an MSP perform routine.

It is common for methods in Max (like a ‘read’ method) to use
defer_low() to call a ‘do_read’ function. The defer* functions are
your friends for pushing stuff back into the low priority thread so
you don’t cause hiccups in timing or audio glitches.

The best place I think I’ve seen Max’s threading explained is in a CMJ
article by Ben Nevile and Randy Jones. Here is a link to the TOC for
that issue: http://www.mitpressjournals.org/toc/comj/29/4 .

> My guess would be that each external method is then called from a
> critical section so that the call can’t be interrupted. But if this
> is the case why shouldn’t you be able to allocate more memory? If
> the problem with allocating more memory is that the memory manager
> isn’t reentrant, then you shouldn’t have any problems allocating
> more memory from a critical section. Maybe, entering interrupt
> level means that an external can be interrupted at any point by
> another thread, but if that’s the case, then it should be no problem
> to allocate more memory using new as long as you call it from a
> critical section.
>
> Hm… Could someone give me some clarification about interrupt
> level please?

In the old MacOS, the software could get direct hardware interrupts.
If Max took too long to computer everything it needed to in that
interrupt then you would be a in a world of hurt. On MacOS X this
isn’t the case. However, doing expensive operations in the high
priority threads will degrade Max’s performance as mentioned above.

As far as how to allocate the memory is concerned,d using "new" is fine.

I hope this helps,
Tim


Viewing 21 posts - 1 through 21 (of 21 total)