resizing a buffer~ in an external


    Jan 19 2008 | 9:19 pm
    hi, i'm trying to write to a buffer in an external i'm making, and this involves resizing it. i'm wondering what the proper way to do this is.
    i've tried something like:
    free(b->b_samples);
    b->b_samples = (float*)malloc(size*sizeof(float));
    and updating the buffers size feilds etc, then just writing to the new b2->b_samples... but i think this is all very wrong. i get memory violation errors when i try to free the old b_samples for starters.
    i'd appreciate any help

    • Jan 19 2008 | 9:29 pm
      pete wrote:
      > hi, i'm trying to write to a buffer in an external i'm making, and this involves resizing it. i'm wondering what the proper way to do this is.
      >
      > i've tried something like:
      >
      > free(b->b_samples);
      > b->b_samples = (float*)malloc(size*sizeof(float));
      You have to use the same memory de-/allocation routines that were used
      to allocate the memory. That's most likely t_getbytes() and
      t_freebytes() but I don't know for sure.
      Additionally, if you change the size you have to set certain values to
      correspond with the new size. Here's a code snippet that shows what I'm
      doing. It seems to work...:
      ----------
      x->x_buffer->b_size = size; // total size in samples (all channels)
      x->x_buffer->b_nchans = x->x_channels;
      x->x_buffer->b_frames = x->x_buffer->b_size / x->x_buffer->b_nchans;
      x->x_buffer->b_outputbytes = sizeof(float);
      ----------
      Olaf
    • Jan 19 2008 | 10:59 pm
      cheers for that, it'll probably work when i can work out what library i need to include for t_getbytes and t_freebytes.. any ideas?
    • Jan 19 2008 | 11:59 pm
      pete wrote:
      > cheers for that, it'll probably work when i can work out what library i need to include for t_getbytes and t_freebytes.. any ideas?
      Same as when writing DSP externals: library is MaxAudio.lib and include
      z_dsp.h.
      Important: Make sure to give your external a name with a ~ so that Max
      knows it's using DSP functions (even if it's just only the memory
      allocation) otherwise your externals would crash in Max Runtime (at
      least that's what used to happen with Max 4.5 on OS X).
      BTW, there is also t_resizebytes() that you can use.
      Olaf
    • Jan 20 2008 | 12:29 am
      thanks! it works now. i still get access violation something on the first t_freebytes call, but only in debug mode so i can live with that.
      i couldnt find documentation anywhere of t_freebytes/getbytes/resizebytes though, which is a bit odd
    • Jan 20 2008 | 12:32 am
      i thought originally it would just be a max external rather than msp, because it has no perform function. the "offline" use of buffer~s is the only msp type aspect of it. should i still give it a ~ (twiddle)?
    • Jan 20 2008 | 8:17 am
    • Jan 20 2008 | 7:59 pm
      t_freebytes() getbytes() resizebytes()
      On Jan 19, 2008 4:29 PM, pete wrote:
      >
      > thanks! it works now. i still get access violation something on the first t_freebytes call, but only in debug mode so i can live with that.
      >
      > i couldnt find documentation anywhere of t_freebytes/getbytes/resizebytes though, which is a bit odd
      >
      >
    • Jan 20 2008 | 9:35 pm
      > hi, i'm trying to write to a buffer in an external i'm making, and this involves resizing it.
      You may find this thread relevant:
      Eric
    • Jan 20 2008 | 9:48 pm
      Quote: bbnickell wrote on Sun, 20 January 2008 20:59
      ----------------------------------------------------
      > t_freebytes() > getbytes() > resizebytes() ----------------------------------------------------
      Not so. getbytes() & Co. look a bit like malloc() & Co., or even like NewPtr() & Co. (the One True Memory Allocation Methods). But they are very, very different. The one is not simply a wrapper for the other. If you mix up the different families of memory allocation methods, you will shoot yourself in your most sensitive anatomy.
      To the original query: the getbytes() routines are documented in the SDK WritingExternals.pdf (where else?). In the version that comes with the 4.5.5 SDK, look at pp. 82, 101-102.
    • Jan 20 2008 | 10:21 pm
      Quote: Olaf Matthes wrote on Sun, 20 January 2008 00:59
      ----------------------------------------------------
      > Important: Make sure to give your external a name with a ~ so that Max
      > knows it's using DSP functions (even if it's just only the memory
      > allocation) otherwise your externals would crash in Max Runtime (at
      > least that's what used to happen with Max 4.5 on OS X).
      ----------------------------------------------------
      Olaf... are you sure about that? It sounds more like urban legend, to be honest.
      In order to use DSP methods, your object needs to call dsp_initclass() inside main(), and its first component needs to be a t_pxobject instead of a t_object. The tilde is simply a naming convention, no more. AFAICT it was primarily instigated to provide a simple way to distinguish between + and +~ (and analogously for about two thousand other object pairs).
      I do recall that at least one C74 quasi-dsp object went through most of its beta incarnation without the tilde, which is why I find the statement that a tilde-less object name would crash the RT a little surprising.
    • Jan 20 2008 | 10:55 pm
      Peter Castine wrote:
      > Quote: Olaf Matthes wrote on Sun, 20 January 2008 00:59
      > ----------------------------------------------------
      >> Important: Make sure to give your external a name with a ~ so that Max
      >> knows it's using DSP functions (even if it's just only the memory
      >> allocation) otherwise your externals would crash in Max Runtime (at
      >> least that's what used to happen with Max 4.5 on OS X).
      > ----------------------------------------------------
      >
      > Olaf... are you sure about that? It sounds more like urban legend, to be honest.
      >
      > In order to use DSP methods, your object needs to call dsp_initclass() inside main(), and its first component needs to be a t_pxobject instead of a t_object. The tilde is simply a naming convention, no more. AFAICT it was primarily instigated to provide a simple way to distinguish between + and +~ (and analogously for about two thousand other object pairs).
      Sorry, it was late at night... when I look at my code now (late at night
      again) I can tell that I used a name with a ~, dsp_initclass() and that
      there is a t_pxobject instead of t_object. So it's basically a DSP
      object without perform routine.
      > I do recall that at least one C74 quasi-dsp object went through most of its beta incarnation without the tilde, which is why I find the statement that a tilde-less object name would crash the RT a little surprising.
      Yes, it was crashing in Runtime when I was calling t_getbytes() and
      friends in a non-DSP external. This was with Max 4.5.? on PPC MacMinis.
      I will meet these machines again in February and can do some more tests.
      It never crashed on my Windows machine (in RT), but on those Macs my
      patch didn't work in Runtime but did work in regular version. After
      making by external a 'pseudo-DSP' object it stopped crashing.
      Olaf
    • Jan 21 2008 | 5:20 am
      The point of the thread I mentioned above is that you really don't need to go through all of this hassle. It's much easier to just send a "size" message to your buffer inside your code, and let Max take care of memory management for you. Although I'm curious if anyone can think of a reason why it would be better to take on memory management for yourself.
      Eric
    • Jan 21 2008 | 9:34 pm
      Quote: Eric Lyon wrote on Mon, 21 January 2008 06:20
      ----------------------------------------------------
      > The point of the thread I mentioned above is that you really don't need to go through all of this hassle. It's much easier to just send a "size" message to your buffer inside your code, and let Max take care of memory management for you. Although I'm curious if anyone can think of a reason why it would be better to take on memory management for yourself.
      ----------------------------------------------------
      Good point.
      My responses were intended to clarify some confusing statements made in the thread, and people really really ought to know *how* to manage memory correctly in Max.
      But, of course, the most direct solution to the OPQ is to sit down with the SDK documentation and read how to use pass messages to Max objects from the C API. And while there, reviewing Max memory management isn't a bad idea, either!
    • Jan 22 2008 | 9:24 am
      void *getbytes(short size)
      void* freebytes(void *ptr, short size)
      The Writing Externals SDK says that this set of memory allocation
      routines are to be used only for interrupt-level memory allocation.
      These are supposed to be reentrant functions capped off with 32767
      bytes entered as short size in which anything bigger than 16384 bytes
      passes allocation off to NewPtr(). NewPtr() is not a reentrant memory
      allocation function and will complain if called at the interrupt level
      by returning size 0. NewPtr() is also specific to Max OSX memory
      allocation so what gets used in the case of interrupt level memory
      allocation greater than 16384 bytes in Win32...something which is also
      like NewPtr or a wrapper to something else? I've never seen this
      brought up and I believe you even asked what the fundamental
      differences are between malloc() and NewPtr() on native-OS memory
      allocation routines:
      The sysmem_* allocation routines outside of interrupt memory
      allocation all have their descriptions more or less in the docs as
      "Like malloc()..." or "(insert native Mac OS allocation routine)" --
      unless I'm completely missing something I haven't seen anything ever
      discussed as to how these sysmem calls are more portable or more safe
      than malloc(), calloc(), realloc() and free(). If anyone can offer a
      technical explanation I would be very much appreciate it as the Memory
      Management section of the SDK doesn't really offer a deeper
      explanation into this particular matter.
      Miller had it down with Pure Data--the functions below had everything
      wrapped around portable malloc routines. The ISO/IEC 9899:1999 ANSI C
      standard ("C API" is a misnomer) says "the functions in the standard
      library are not guaranteed to be reentrant and may modify objects with
      static storage duration" so it's left to the compiler implementation
      whether or not to implement C library functions as reentrant or not.
      Perhaps it's this reason that the traditional getbytes/freebytes
      functions differ in their parameters with the Max API by limiting it
      to a set of reentrant routines with a short type. Pure Data
      getbytes()/freebytes():
      /* Copyright (c) 1997-1999 Miller Puckette.
      * For information on usage and redistribution, and for a DISCLAIMER OF ALL
      * WARRANTIES, see the file, "LICENSE.txt," in this distribution. */
      void *getbytes(size_t nbytes)
      {
      void *ret;
      if (nbytes < 1) nbytes = 1;
      ret = (void *)calloc(nbytes, 1);
      #ifdef LOUD
      fprintf(stderr, "new %lx %dn", (int)ret, nbytes);
      #endif /* LOUD */
      #ifdef DEBUGMEM
      totalmem += nbytes;
      #endif
      if (!ret)
      post("pd: getbytes() failed -- out of memory");
      return (ret);
      }
      void *resizebytes(void *old, size_t oldsize, size_t newsize)
      {
      void *ret;
      if (newsize < 1) newsize = 1;
      if (oldsize < 1) oldsize = 1;
      ret = (void *)realloc((char *)old, newsize);
      if (newsize > oldsize && ret)
      memset(((char *)ret) + oldsize, 0, newsize - oldsize);
      #ifdef LOUD
      fprintf(stderr, "resize %lx %d --> %lx %dn", (int)old, oldsize,
      (int)ret, newsize);
      #endif /* LOUD */
      #ifdef DEBUGMEM
      totalmem += (newsize - oldsize);
      #endif
      if (!ret)
      post("pd: resizebytes() failed -- out of memory");
      return (ret);
      }
      void freebytes(void *fatso, size_t nbytes)
      {
      if (nbytes == 0)
      nbytes = 1;
      #ifdef LOUD
      fprintf(stderr, "free %lx %dn", (int)fatso, nbytes);
      #endif /* LOUD */
      #ifdef DEBUGMEM
      totalmem -= nbytes;
      #endif
      free(fatso);
      }
      On Jan 21, 2008 1:34 PM, Peter Castine wrote:
      >
      > Quote: Eric Lyon wrote on Mon, 21 January 2008 06:20
      > ----------------------------------------------------
      > > The point of the thread I mentioned above is that you really don't need to go through all of this hassle. It's much easier to just send a "size" message to your buffer inside your code, and let Max take care of memory management for you. Although I'm curious if anyone can think of a reason why it would be better to take on memory management for yourself.
      > ----------------------------------------------------
      >
      > Good point.
      >
      > My responses were intended to clarify some confusing statements made in the thread, and people really really ought to know *how* to manage memory correctly in Max.
      >
      > But, of course, the most direct solution to the OPQ is to sit down with the SDK documentation and read how to use pass messages to Max objects from the C API. And while there, reviewing Max memory management isn't a bad idea, either!
      > --
      > Peter Castine
      > --
      > Peter is in dire need of a new Facebook tagline. Not to mention a new .sig
      >
      >
    • Jan 22 2008 | 12:25 pm
      back to the original question in case anyone is interested, i'm sending buffer~ a size message now like eric suggested and all is good. i'm used to higher level OO, so it does seem kind of wrong to me to be doing memory management across objects.
      thanks for all the help
    • Jan 22 2008 | 12:52 pm
      > In order to use DSP methods, your object needs to call dsp_initclass() inside main(), and its first component needs to be a t_pxobject instead of a t_object. The tilde is simply a naming convention, no more. AFAICT it was primarily instigated to provide a simple way to distinguish between + and +~ (and analogously for about two thousand other object pairs).
      i tried changing my object's first component to t_pxobject and adding a call to dsp_initclass() in main(), because i gathered this is the right thing to do as my object uses buffer~ and should therefore have a ~ itself. but doing this made max crash whenever i turned dsp off. also, i noticed that buffer~ itself does not have a t_pxobject as its first component.
      so, what i'm trying to say is that my object does not have a perform routine, doesnt call dsp_initclass() and does not have a t_pxobject, so should it realy have a ~? not that it matters much, i'm just curious about conventions.
    • Jan 22 2008 | 2:27 pm
      Quote: bbnickell wrote on Tue, 22 January 2008 10:24
      ----------------------------------------------------
      > void *getbytes(short size)
      > void* freebytes(void *ptr, short size)
      >
      > The Writing Externals SDK says that this set of memory allocation
      > routines are to be used only for interrupt-level memory allocation.
      ----------------------------------------------------
      I don't see the word "only" in the documentation. And lots of existing code uses the getbytes() family exclusively, regardless of interrupt concerns.
      FTR, I wrote a couple of externals that tested for isr(), used native memory allocation if safe and getbytes() otherwise, stored a flag to note which kind of allocation was used so that the proper deallocation would be called, and even deferred deallocation in case that would be necessary.
      All of which was pretty much a waste of effort on my part.
      My rule of thumb: Use native memory allocation for big stuff. Use getbytes() for small stuff. Prends-le ou laisse.
      Note that Miller wrote in his 1980s papers on developing Max that he developed getbytes() & Co. to be particularly efficient. And compared to the contemporaneous malloc() implementations, he succeeded. I'll have to dig up the ref in case anyone cares.
      It's all well and good to say what Pd does things much better than Max/MSP, but Pd does not have to be compatible with 20 years of Max patches & externals. I cannot say whether this is an issue in this particular case, but it is worth taking note of.
      --
      Regarding cross-platform issues. Getbytes() has the advantage that you don't have to worry about host OS. Alternatives for native memory allocation are:
      - sysmem_newptr() & Co.: wrapper functions to whatever the host malloc() is. Use like malloc() & co. Only drawback: overhead for an extra procedure call (setting up stack etc.) Advantage: you don't have to worry about the host OS.
      - Conditionally compile
      #if WIN_VERSION
      #define NewPtr(x) malloc(x)
      // etc. etc. etc.
      #endif
      Quick, and only requires a few extra lines of code. You could also #if... #define in the other direction. All a question of taste. Since OS X this isn't absolutely necessary, because you can simply malloc() on either Mac or Windows. You can even malloc() in interrupt nowadays, it's just that malloc() may take more time than you want to wait in the middle of, say, an MSP perform() routine.
      Hope this helps,
      P.
    • Jan 22 2008 | 2:39 pm
      Quote: peterworth@gmail.com wrote on Tue, 22 January 2008 13:52
      ----------------------------------------------------
      > i tried changing my object's first component to t_pxobject and adding a call to dsp_initclass() in main(), because i gathered this is the right thing to do as my object uses buffer~ and should therefore have a ~ itself. but doing this made max crash whenever i turned dsp off. also, i noticed that buffer~ itself does not have a t_pxobject as its first component.
      >
      > so, what i'm trying to say is that my object does not have a perform routine, doesnt call dsp_initclass() and does not have a t_pxobject, so should it realy have a ~? not that it matters much, i'm just curious about conventions.
      ----------------------------------------------------
      I think buffer~ & co. answer your second paragraph.
      For the first paragraph: hard to say without seeing code.
      Again: the tilde is a convention. AFAICT, the Max engine does *not* look at the object name, strchr() on '~' and then decide whether or not to do striped patch cords. It is the call to dsp_initclass() that does the magic.
    • Jan 31 2008 | 10:35 am
      I've been reading through the Max external developers manual trying to understand whether or not it's possible to allocate more memory in Max using new (in C++) while audio processing is running, but I'm having a fair amount of difficulty understanding the manual despite reading through it several times.
      The manual talks about running at interrupt level for instance, and apparently, you enter interrupt level when you put Max in overdrive. Do you also enter interrupt level when you turn on audio processing in MSP? And what exactly is interrupt level? My guess would be that each external method is then called from a critical section so that the call can't be interrupted. But if this is the case why shouldn't you be able to allocate more memory? If the problem with allocating more memory is that the memory manager isn't reentrant, then you shouldn't have any problems allocating more memory from a critical section. Maybe, entering interrupt level means that an external can be interrupted at any point by another thread, but if that's the case, then it should be no problem to allocate more memory using new as long as you call it from a critical section.
      Hm... Could someone give me some clarification about interrupt level please?
      Best,
      Greg
    • Feb 01 2008 | 12:55 pm
      On 2008 Jan 31, at 4:35 AM, Greg Kellum wrote:
      > I've been reading through the Max external developers manual trying
      > to understand whether or not it's possible to allocate more memory
      > in Max using new (in C++) while audio processing is running, but I'm
      > having a fair amount of difficulty understanding the manual despite
      > reading through it several times.
      >
      > The manual talks about running at interrupt level for instance, and
      > apparently, you enter interrupt level when you put Max in overdrive.
      > Do you also enter interrupt level when you turn on audio processing
      > in MSP? And what exactly is interrupt level?
      Hi,
      Basically, you want to avoid doing expensive operations (like memory
      allocation and deallocation) in anything called directly by the
      scheduler or the in an MSP perform routine.
      It is common for methods in Max (like a 'read' method) to use
      defer_low() to call a 'do_read' function. The defer* functions are
      your friends for pushing stuff back into the low priority thread so
      you don't cause hiccups in timing or audio glitches.
      The best place I think I've seen Max's threading explained is in a CMJ
      article by Ben Nevile and Randy Jones. Here is a link to the TOC for
      > My guess would be that each external method is then called from a
      > critical section so that the call can't be interrupted. But if this
      > is the case why shouldn't you be able to allocate more memory? If
      > the problem with allocating more memory is that the memory manager
      > isn't reentrant, then you shouldn't have any problems allocating
      > more memory from a critical section. Maybe, entering interrupt
      > level means that an external can be interrupted at any point by
      > another thread, but if that's the case, then it should be no problem
      > to allocate more memory using new as long as you call it from a
      > critical section.
      >
      > Hm... Could someone give me some clarification about interrupt
      > level please?
      In the old MacOS, the software could get direct hardware interrupts.
      If Max took too long to computer everything it needed to in that
      interrupt then you would be a in a world of hurt. On MacOS X this
      isn't the case. However, doing expensive operations in the high
      priority threads will degrade Max's performance as mentioned above.
      As far as how to allocate the memory is concerned,d using "new" is fine.
      I hope this helps,
      Tim