Forums > Dev

Crashing probably due to memory limits

November 27, 2007 | 9:29 am

Hi all,

I’m making an MSP external which some times works quite fine, but some
times it crashes, and I simply don’t know why. As a very short
description, it has a huge array of floats (a lookup table) which may
change by sending some parameters to the object’s inlet. The array’s
size changes permanently, but I implemented a mechanism that allocates
new memory only if the size is growing. The object’s behaviour is: if I
don’t change anything, or if I only do some slow changes in the
parameters, everything works OK, but if I start to change the parameters
very fast (for example, by clicking on one of the number boxes connected
to the inlets and moving the mouse very fast for seconds) I get an
exception EXC_BAD_ACCESS (code KERN_INVALID_ADDRESS). I’m suspecting
that this should be due to some memory management error (the Activity
Monitor says that I’m using more than 1GB of virtual and about 400-500
MB of physical), but I don’t know how to be sure about this. Anyway, can
it happen that the sysmem_newptrclear can’t allocate me more memory for
some reason? (I have 2 GB RAM, so I should have more memory in my
machine.) Is there a difference in using sysmem_newptrclear or simply
using malloc (I know there are differences, but I don’t know if they
affect the maximal amount of memory that I could get)?
One more thing that I don’t know, but may cause the problem: when do the
‘list’ message handlers execute in a DPS object? Is it possible that
during an execution of the perform routine the list handler is being
executed in an other thread? (If yes, this is surely the root of my
problem, as I’m using the lookup table in my perform routine, so it
shouldn’t change during the execution of the perform routine.)

Thank you,
Adam


November 27, 2007 | 9:37 am

Are you freeing your old pointer before you allocate a new one? There
are also:

extern t_ptr sysmem_resizeptr(void *ptr, long newsize);
extern t_ptr sysmem_resizeptrclear(void *ptr, long newsize);

which take care of the destruction and creation of the new pointer
for you + copy the old data.

jb

Am 27.11.2007 um 10:29 schrieb Siska Ádám:

> Hi all,
>
>
> I’m making an MSP external which some times works quite fine, but
> some times it crashes, and I simply don’t know why. As a very short
> description, it has a huge array of floats (a lookup table) which
> may change by sending some parameters to the object’s inlet. The
> array’s size changes permanently, but I implemented a mechanism
> that allocates new memory only if the size is growing. The object’s
> behaviour is: if I don’t change anything, or if I only do some slow
> changes in the parameters, everything works OK, but if I start to
> change the parameters very fast (for example, by clicking on one of
> the number boxes connected to the inlets and moving the mouse very
> fast for seconds) I get an exception EXC_BAD_ACCESS (code
> KERN_INVALID_ADDRESS). I’m suspecting that this should be due to
> some memory management error (the Activity Monitor says that I’m
> using more than 1GB of virtual and about 400-500 MB of physical),
> but I don’t know how to be sure about this. Anyway, can it happen
> that the sysmem_newptrclear can’t allocate me more memory for some
> reason? (I have 2 GB RAM, so I should have more memory in my
> machine.) Is there a difference in using sysmem_newptrclear or
> simply using malloc (I know there are differences, but I don’t know
> if they affect the maximal amount of memory that I could get)?
> One more thing that I don’t know, but may cause the problem: when
> do the ‘list’ message handlers execute in a DPS object? Is it
> possible that during an execution of the perform routine the list
> handler is being executed in an other thread? (If yes, this is
> surely the root of my problem, as I’m using the lookup table in my
> perform routine, so it shouldn’t change during the execution of the
> perform routine.)
>
> Thank you,
> Adam
>


November 27, 2007 | 5:10 pm

Hi,

yes, I do. I tried to avoid the re-allocation of the pointer as much as
possible, and at that few parts I free everything before allocating a
new pointer.
The sysmem_resizeptr sounds like a good solution, but I should know
something about the way it works: how is it expected to perform if I
call it very-very often (maybe after every execution of the perform
routine)? Does it allocate a new area for the resized pointer every
time, or does it have some optimization for these cases?

Thank you,
Adam

________________
Siska Ádám
+36 (70) 207-63-85

http://apocalypse.rulez.org/~sadam

Jeremy Bernstein wrote:
> Are you freeing your old pointer before you allocate a new one? There
> are also:
>
> extern t_ptr sysmem_resizeptr(void *ptr, long newsize);
> extern t_ptr sysmem_resizeptrclear(void *ptr, long newsize);
>
> which take care of the destruction and creation of the new pointer for
> you + copy the old data.
>
> jb
>
> Am 27.11.2007 um 10:29 schrieb Siska Ádám:
>
>> Hi all,
>>
>>
>> I’m making an MSP external which some times works quite fine, but
>> some times it crashes, and I simply don’t know why. As a very short
>> description, it has a huge array of floats (a lookup table) which may
>> change by sending some parameters to the object’s inlet. The array’s
>> size changes permanently, but I implemented a mechanism that
>> allocates new memory only if the size is growing. The object’s
>> behaviour is: if I don’t change anything, or if I only do some slow
>> changes in the parameters, everything works OK, but if I start to
>> change the parameters very fast (for example, by clicking on one of
>> the number boxes connected to the inlets and moving the mouse very
>> fast for seconds) I get an exception EXC_BAD_ACCESS (code
>> KERN_INVALID_ADDRESS). I’m suspecting that this should be due to some
>> memory management error (the Activity Monitor says that I’m using
>> more than 1GB of virtual and about 400-500 MB of physical), but I
>> don’t know how to be sure about this. Anyway, can it happen that the
>> sysmem_newptrclear can’t allocate me more memory for some reason? (I
>> have 2 GB RAM, so I should have more memory in my machine.) Is there
>> a difference in using sysmem_newptrclear or simply using malloc (I
>> know there are differences, but I don’t know if they affect the
>> maximal amount of memory that I could get)?
>> One more thing that I don’t know, but may cause the problem: when do
>> the ‘list’ message handlers execute in a DPS object? Is it possible
>> that during an execution of the perform routine the list handler is
>> being executed in an other thread? (If yes, this is surely the root
>> of my problem, as I’m using the lookup table in my perform routine,
>> so it shouldn’t change during the execution of the perform routine.)
>>
>> Thank you,
>> Adam
>>
>
>
>


November 27, 2007 | 9:57 pm

Quote: sadam wrote on Tue, 27 November 2007 10:10
—————————————————-
> Hi,
>
>
> yes, I do. I tried to avoid the re-allocation of the pointer as much as
> possible, and at that few parts I free everything before allocating a
> new pointer.

This may (as you point out above) be your problem. If the pointer is being freed in a scheduler thread and the perform routine is interrupting it then it is possible that the perform routine is called at the point where the pointer has been freed, but the new memory not yet allocated – at this point you’ll get a crash (if you check the crash log you should be able to see if it is a call to the lookup table that is crashing – find the thread that is crashing then look at the routines crashing at see what line of your code throws the exception). If you allocate new memory to a temporary pointer, then replace the pointer with the newly allocated one, and finally free the old pointer you may solve this problem…..

Regards,

Alex


November 27, 2007 | 9:58 pm

Quote: sadam wrote on Tue, 27 November 2007 10:10
—————————————————-
> Hi,
>
>
> yes, I do. I tried to avoid the re-allocation of the pointer as much as
> possible, and at that few parts I free everything before allocating a
> new pointer.

This may (as you point out above) be your problem. If the pointer is being freed in a scheduler thread and the perform routine is interrupting it then it is possible that the perform routine is called at the point where the pointer has been freed, but the new memory not yet allocated – at this point you’ll get a crash (if you check the crash log you should be able to see if it is a call to the lookup table that is crashing – find the thread that is crashing then look at the routines crashing at see what line of your code throws the exception). If you allocate new memory to a temporary pointer, copy the data, then replace the pointer with the newly allocated one, and finally free the old pointer you may solve this problem…..

Regards,

Alex


November 27, 2007 | 10:00 pm

Apologies for the almost identical posts – I missed a step in the first one and was locked out from editing by the time I’d realised. If a moderator wished to delete the first post that would be handy.

Alex


November 27, 2007 | 11:44 pm


November 28, 2007 | 12:07 am

I think maybe I wasn’t clear before. The problem seems to be that you are freeing the pointer before you allocate a new one leading the possibility of the perform routine getting an invalid pointer. You could copy it and free it after allocating new memory (and doing some other stuff – see below) instead in order to avoid this situation.

Quote: sadam wrote on Tue, 27 November 2007 16:44
—————————————————-
> Hello,
>
>
> Is there a possibility to force the ‘list’ message event handler to run
> only between the execution of two subsequent perform routines?
>

I think maybe not. I don’t know for sure, but anyway it will be the audio thread interrupting the scheduler thread and not the other way round I believe, so the scheduler list routine will only start running when audio is not processing. The problem arises when the audio processing stops the list routine midway through. I’m suggesting you do the following in your list routine to fix the problem:

1 – assign new memory to temp pointer
2 – copy data to new pointer + add any necessary extra data
3 – once done copy old pointer from object to second temp location.
4 – replace pointer in object with new one from temp
5 – update object info about size of pointer (for perform routine)
6 – free old memory using second temp pointer

This way your perform routine always gets a valid pointer with valid data. There is also no need to check the validity of the pointer, which you probably don’t want to do anyway as I’d imagine your perform routine won’t be able to function correctly without a valid pointer.

Also if you’re resizing a lot I’d rethink how much you grow the pointer or the initial size allocated. If the above doesn’t solve it then maybe you have a different problem.

Alex


November 28, 2007 | 1:24 am


November 28, 2007 | 2:25 am

Quote: sadam wrote on Tue, 27 November 2007 18:24
—————————————————-
> Hello,
>
>
> Well, I’m doing exactly the mechanism you described. Actually, I created
> a garbage collector object, so the freeing of the pointers are being
> executed only at the end of the respective routines (this ‘garbage
> collector’ has only a collection of the pointers that should be
> deallocated, and the destructor of the object iterates through these
> pointers and deallocates the pointers). So the variables in my object
> can’t became invalid. But I’m doing some loops like….

OK I’ve done some quick tests – looks like if you’re not running the scheduler in audio interrupt and you have a multicore processor then the threads could be running at the same time with either routine starting first. This probably means a more complex garbage collection routine where you check if you’re in the perform routine (store a flag in your object that is set in entry/exit). If you aren’t then you are good to free the resources, Otherwise you need to defer until the perform routine has finished somehow. This could be a bit tricky – not sure of the best way, but one solution would be:

1 – Maintain a list of any pointers used by the perform routine that could not be freed immediately, each with a not_safe flag set to 1.
2 – At the end of your perform routine examine the list and set any current members not_safe flags to 0 (the perform routine will use the new pointers for these next time round)
3 – setup a clock that is called in the new routine and calls itself every n milliseconds – in your clock routine check your list of pointers and if they are marked as safe free them, otherwise leave them there.

Of course you need a dynamic list of pointers as it’s possible that when you defer then the list function has been called again and there is more memory to be freed, although you really shouldn’t need to resize that often, and you could always lock the list routine out yourself until garbage collection has been done if you wsnt to limit the possibility of that happening

Alex


November 28, 2007 | 3:19 am


Viewing 11 posts - 1 through 11 (of 11 total)