$adam

I'm making an MSP external which some times works quite fine, but some 

times it crashes, and I simply don't know why. As a very short 

description, it has a huge array of floats (a lookup table) which may 

change by sending some parameters to the object's inlet. The array's 

size changes permanently, but I implemented a mechanism that allocates 

new memory only if the size is growing. The object's behaviour is: if I 

don't change anything, or if I only do some slow changes in the 

parameters, everything works OK, but if I start to change the parameters 

very fast (for example, by clicking on one of the number boxes connected 

to the inlets and moving the mouse very fast for seconds) I get an 

exception EXC_BAD_ACCESS (code KERN_INVALID_ADDRESS). I'm suspecting 

that this should be due to some memory management error (the Activity 

Monitor says that I'm using more than 1GB of virtual and about 400-500 

MB of physical), but I don't know how to be sure about this. Anyway, can 

it happen that the sysmem_newptrclear can't allocate me more memory for 

some reason? (I have 2 GB RAM, so I should have more memory in my 

machine.) Is there a difference in using sysmem_newptrclear or simply 

using malloc (I know there are differences, but I don't know if they 

affect the maximal amount of memory that I could get)?

One more thing that I don't know, but may cause the problem: when do the 

'list' message handlers execute in a DPS object? Is it possible that 

during an execution of the perform routine the list handler is being 

executed in an other thread? (If yes, this is surely the root of my 

problem, as I'm using the lookup table in my perform routine, so it 

shouldn't change during the execution of the perform routine.)

I'm making an MSP external which some times works quite fine, but some 
times it crashes, and I simply don't know why. As a very short 
description, it has a huge array of floats (a lookup table) which may 
change by sending some parameters to the object's inlet. The array's 
size changes permanently, but I implemented a mechanism that allocates 
new memory only if the size is growing. The object's behaviour is: if I 
don't change anything, or if I only do some slow changes in the 
parameters, everything works OK, but if I start to change the parameters 
very fast (for example, by clicking on one of the number boxes connected 
to the inlets and moving the mouse very fast for seconds) I get an 
exception EXC_BAD_ACCESS (code KERN_INVALID_ADDRESS). I'm suspecting 
that this should be due to some memory management error (the Activity 
Monitor says that I'm using more than 1GB of virtual and about 400-500 
MB of physical), but I don't know how to be sure about this. Anyway, can 
it happen that the sysmem_newptrclear can't allocate me more memory for 
some reason? (I have 2 GB RAM, so I should have more memory in my 
machine.) Is there a difference in using sysmem_newptrclear or simply 
using malloc (I know there are differences, but I don't know if they 
affect the maximal amount of memory that I could get)?
One more thing that I don't know, but may cause the problem: when do the 
'list' message handlers execute in a DPS object? Is it possible that 
during an execution of the perform routine the list handler is being 
executed in an other thread? (If yes, this is surely the root of my 
problem, as I'm using the lookup table in my perform routine, so it 
shouldn't change during the execution of the perform routine.)

crashing-probably-due-to-memory-limits

Are you freeing your old pointer before you allocate a new one? There 

extern t_ptr sysmem_resizeptr(void *ptr, long newsize);

extern t_ptr sysmem_resizeptrclear(void *ptr, long newsize);

which take care of the destruction and creation of the new pointer 

Am 27.11.2007 um 10:29 schrieb Siska Ádám:

> I'm making an MSP external which some times works quite fine, but 

> some times it crashes, and I simply don't know why. As a very short 

> description, it has a huge array of floats (a lookup table) which 

> may change by sending some parameters to the object's inlet. The 

> array's size changes permanently, but I implemented a mechanism 

> that allocates new memory only if the size is growing. The object's 

> behaviour is: if I don't change anything, or if I only do some slow 

> changes in the parameters, everything works OK, but if I start to 

> change the parameters very fast (for example, by clicking on one of 

> the number boxes connected to the inlets and moving the mouse very 

> fast for seconds) I get an exception EXC_BAD_ACCESS (code 

> KERN_INVALID_ADDRESS). I'm suspecting that this should be due to 

> some memory management error (the Activity Monitor says that I'm 

> using more than 1GB of virtual and about 400-500 MB of physical), 

> but I don't know how to be sure about this. Anyway, can it happen 

> that the sysmem_newptrclear can't allocate me more memory for some 

> reason? (I have 2 GB RAM, so I should have more memory in my 

> machine.) Is there a difference in using sysmem_newptrclear or 

> simply using malloc (I know there are differences, but I don't know 

> if they affect the maximal amount of memory that I could get)?

> One more thing that I don't know, but may cause the problem: when 

> do the 'list' message handlers execute in a DPS object? Is it 

> possible that during an execution of the perform routine the list 

> handler is being executed in an other thread? (If yes, this is 

> surely the root of my problem, as I'm using the lookup table in my 

> perform routine, so it shouldn't change during the execution of the 

Are you freeing your old pointer before you allocate a new one? There  
are also:

extern t_ptr sysmem_resizeptr(void *ptr, long newsize);
extern t_ptr sysmem_resizeptrclear(void *ptr, long newsize);

which take care of the destruction and creation of the new pointer  
for you + copy the old data.

> Hi all,
>
>
> I'm making an MSP external which some times works quite fine, but  
> some times it crashes, and I simply don't know why. As a very short  
> description, it has a huge array of floats (a lookup table) which  
> may change by sending some parameters to the object's inlet. The  
> array's size changes permanently, but I implemented a mechanism  
> that allocates new memory only if the size is growing. The object's  
> behaviour is: if I don't change anything, or if I only do some slow  
> changes in the parameters, everything works OK, but if I start to  
> change the parameters very fast (for example, by clicking on one of  
> the number boxes connected to the inlets and moving the mouse very  
> fast for seconds) I get an exception EXC_BAD_ACCESS (code  
> KERN_INVALID_ADDRESS). I'm suspecting that this should be due to  
> some memory management error (the Activity Monitor says that I'm  
> using more than 1GB of virtual and about 400-500 MB of physical),  
> but I don't know how to be sure about this. Anyway, can it happen  
> that the sysmem_newptrclear can't allocate me more memory for some  
> reason? (I have 2 GB RAM, so I should have more memory in my  
> machine.) Is there a difference in using sysmem_newptrclear or  
> simply using malloc (I know there are differences, but I don't know  
> if they affect the maximal amount of memory that I could get)?
> One more thing that I don't know, but may cause the problem: when  
> do the 'list' message handlers execute in a DPS object? Is it  
> possible that during an execution of the perform routine the list  
> handler is being executed in an other thread? (If yes, this is  
> surely the root of my problem, as I'm using the lookup table in my  
> perform routine, so it shouldn't change during the execution of the  
> perform routine.)
>
> Thank you,
> Adam
>

yes, I do. I tried to avoid the re-allocation of the pointer as much as 

possible, and at that few parts I free everything before allocating a 

The sysmem_resizeptr sounds like a good solution, but I should know 

something about the way it works: how is it expected to perform if I 

call it very-very often (maybe after every execution of the perform 

routine)? Does it allocate a new area for the resized pointer every 

time, or does it have some optimization for these cases?

> Are you freeing your old pointer before you allocate a new one? There 

> extern t_ptr sysmem_resizeptr(void *ptr, long newsize);

> extern t_ptr sysmem_resizeptrclear(void *ptr, long newsize);

> which take care of the destruction and creation of the new pointer for 

> Am 27.11.2007 um 10:29 schrieb Siska Ádám:

>> I'm making an MSP external which some times works quite fine, but 

>> some times it crashes, and I simply don't know why. As a very short 

>> description, it has a huge array of floats (a lookup table) which may 

>> change by sending some parameters to the object's inlet. The array's 

>> size changes permanently, but I implemented a mechanism that 

>> allocates new memory only if the size is growing. The object's 

>> behaviour is: if I don't change anything, or if I only do some slow 

>> changes in the parameters, everything works OK, but if I start to 

>> change the parameters very fast (for example, by clicking on one of 

>> the number boxes connected to the inlets and moving the mouse very 

>> fast for seconds) I get an exception EXC_BAD_ACCESS (code 

>> KERN_INVALID_ADDRESS). I'm suspecting that this should be due to some 

>> memory management error (the Activity Monitor says that I'm using 

>> more than 1GB of virtual and about 400-500 MB of physical), but I 

>> don't know how to be sure about this. Anyway, can it happen that the 

>> sysmem_newptrclear can't allocate me more memory for some reason? (I 

>> have 2 GB RAM, so I should have more memory in my machine.) Is there 

>> a difference in using sysmem_newptrclear or simply using malloc (I 

>> know there are differences, but I don't know if they affect the 

>> maximal amount of memory that I could get)?

>> One more thing that I don't know, but may cause the problem: when do 

>> the 'list' message handlers execute in a DPS object? Is it possible 

>> that during an execution of the perform routine the list handler is 

>> being executed in an other thread? (If yes, this is surely the root 

>> of my problem, as I'm using the lookup table in my perform routine, 

>> so it shouldn't change during the execution of the perform routine.)

yes, I do. I tried to avoid the re-allocation of the pointer as much as 
possible, and at that few parts I free everything before allocating a 
new pointer.
The sysmem_resizeptr sounds like a good solution, but I should know 
something about the way it works: how is it expected to perform if I 
call it very-very often (maybe after every execution of the perform 
routine)? Does it allocate a new area for the resized pointer every 
time, or does it have some optimization for these cases?

________________
Siska Ádám
+36 (70) 207-63-85
http://apocalypse.rulez.org/~sadam

Jeremy Bernstein wrote:
> Are you freeing your old pointer before you allocate a new one? There 
> are also:
>
> extern t_ptr sysmem_resizeptr(void *ptr, long newsize);
> extern t_ptr sysmem_resizeptrclear(void *ptr, long newsize);
>
> which take care of the destruction and creation of the new pointer for 
> you + copy the old data.
>
> jb
>
> Am 27.11.2007 um 10:29 schrieb Siska Ádám:
>
>> Hi all,
>>
>>
>> I'm making an MSP external which some times works quite fine, but 
>> some times it crashes, and I simply don't know why. As a very short 
>> description, it has a huge array of floats (a lookup table) which may 
>> change by sending some parameters to the object's inlet. The array's 
>> size changes permanently, but I implemented a mechanism that 
>> allocates new memory only if the size is growing. The object's 
>> behaviour is: if I don't change anything, or if I only do some slow 
>> changes in the parameters, everything works OK, but if I start to 
>> change the parameters very fast (for example, by clicking on one of 
>> the number boxes connected to the inlets and moving the mouse very 
>> fast for seconds) I get an exception EXC_BAD_ACCESS (code 
>> KERN_INVALID_ADDRESS). I'm suspecting that this should be due to some 
>> memory management error (the Activity Monitor says that I'm using 
>> more than 1GB of virtual and about 400-500 MB of physical), but I 
>> don't know how to be sure about this. Anyway, can it happen that the 
>> sysmem_newptrclear can't allocate me more memory for some reason? (I 
>> have 2 GB RAM, so I should have more memory in my machine.) Is there 
>> a difference in using sysmem_newptrclear or simply using malloc (I 
>> know there are differences, but I don't know if they affect the 
>> maximal amount of memory that I could get)?
>> One more thing that I don't know, but may cause the problem: when do 
>> the 'list' message handlers execute in a DPS object? Is it possible 
>> that during an execution of the perform routine the list handler is 
>> being executed in an other thread? (If yes, this is surely the root 
>> of my problem, as I'm using the lookup table in my perform routine, 
>> so it shouldn't change during the execution of the perform routine.)
>>
>> Thank you,
>> Adam
>>
>
>
>


Quote: sadam wrote on Tue, 27 November 2007 10:10

----------------------------------------------------

> yes, I do. I tried to avoid the re-allocation of the pointer as much as 

> possible, and at that few parts I free everything before allocating a 

This may (as you point out above) be your problem. If the pointer is being freed in a scheduler thread and the perform routine is interrupting it then it is possible that the perform routine is called at the point where the pointer has been freed, but the new memory not yet allocated - at this point you'll get a crash (if you check the crash log you should be able to see if it is a call to the lookup table that is crashing - find the thread that is crashing then look at the routines crashing at see what line of your code throws the exception). If you allocate new memory to a temporary pointer, then replace the pointer with the newly allocated one, and finally free the old pointer you may solve this problem.....

Quote: sadam wrote on Tue, 27 November 2007 10:10
----------------------------------------------------
> Hi,
>
>
> yes, I do. I tried to avoid the re-allocation of the pointer as much as 
> possible, and at that few parts I free everything before allocating a 
> new pointer.

This may (as you point out above) be your problem. If the pointer is being freed in a scheduler thread and the perform routine is interrupting it then it is possible that the perform routine is called at the point where the pointer has been freed, but the new memory not yet allocated - at this point you'll get a crash (if you check the crash log you should be able to see if it is a call to the lookup table that is crashing - find the thread that is crashing then look at the routines crashing at see what line of your code throws the exception). If you allocate new memory to a temporary pointer, copy the data, then replace the pointer with the newly allocated one, and finally free the old pointer you may solve this problem.....

Apologies for the almost identical posts - I missed a step in the first one and was locked out from editing by the time I'd realised. If a moderator wished to delete the first post that would be handy.

I think maybe I wasn't clear before. The problem seems to be that you are freeing the pointer before you allocate a new one leading the possibility of the perform routine getting an invalid pointer. You could copy it and free it after allocating new memory (and doing some other stuff - see below) instead in order to avoid this situation.

Quote: sadam wrote on Tue, 27 November 2007 16:44

> Is there a possibility to force the 'list' message event handler to run 

> only between the execution of two subsequent perform routines?

I think maybe not. I don't know for sure, but anyway it will be the audio thread interrupting the scheduler thread and not the other way round I believe, so the scheduler list routine will only start running when audio is not processing. The problem arises when the audio processing stops the list routine midway through. I'm suggesting you do the following in your list routine to fix the problem:

2 - copy data to new pointer + add any necessary extra data

3 - once done copy old pointer from object to second temp location.

4 - replace pointer in object with new one from temp

5 - update object info about size of pointer (for perform routine)

6 - free old memory using second temp pointer

This way your perform routine always gets a valid pointer with valid data. There is also no need to check the validity of the pointer, which you probably don't want to do anyway as I'd imagine your perform routine won't be able to function correctly without a valid pointer.

Also if you're resizing a lot I'd rethink how much you grow the pointer or the initial size allocated. If the above doesn't solve it then maybe you have a different problem.

Quote: sadam wrote on Tue, 27 November 2007 16:44
----------------------------------------------------
> Hello,
>
>
> Is there a possibility to force the 'list' message event handler to run 
> only between the execution of two subsequent perform routines?
>

1 - assign new memory to temp pointer
2 - copy data to new pointer + add any necessary extra data
3 - once done copy old pointer from object to second temp location.
4 - replace pointer in object with new one from temp
5 - update object info about size of pointer (for perform routine)
6 - free old memory using second temp pointer

Quote: sadam wrote on Tue, 27 November 2007 18:24

> Well, I'm doing exactly the mechanism you described. Actually, I created 

> a garbage collector object, so the freeing of the pointers are being 

> executed only at the end of the respective routines (this 'garbage 

> collector' has only a collection of the pointers that should be 

> deallocated, and the destructor of the object iterates through these 

> pointers and deallocates the pointers). So the variables in my object 

> can't became invalid. But I'm doing some loops like....

OK I've done some quick tests - looks like if you're not running the scheduler in audio interrupt and you have a multicore processor then the threads could be running at the same time with either routine starting first. This probably means a more complex garbage collection routine where you check if you're in the perform routine (store a flag in your object that is set in entry/exit). If you aren't then you are good to free the resources, Otherwise you need to defer until the perform routine has finished somehow. This could be a bit tricky - not sure of the best way, but one solution would be:

1 - Maintain a list of any pointers used by the perform routine that could not be freed immediately, each with a not_safe flag set to 1.

2 - At the end of your perform routine examine the list and set any current members not_safe flags to 0 (the perform routine will use the new pointers for these next time round)

3 - setup a clock that is called in the new routine and calls itself every n milliseconds - in your clock routine check your list of pointers and if they are marked as safe free them, otherwise leave them there.

Of course you need a dynamic list of pointers as it's possible that when you defer then the list function has been called again and there is more memory to be freed, although you really shouldn't need to resize that often, and you could always lock the list routine out yourself until garbage collection has been done if you wsnt to limit the possibility of that happening

Quote: sadam wrote on Tue, 27 November 2007 18:24
----------------------------------------------------
> Hello,
>
>
> Well, I'm doing exactly the mechanism you described. Actually, I created 
> a garbage collector object, so the freeing of the pointers are being 
> executed only at the end of the respective routines (this 'garbage 
> collector' has only a collection of the pointers that should be 
> deallocated, and the destructor of the object iterates through these 
> pointers and deallocates the pointers). So the variables in my object 
> can't became invalid. But I'm doing some loops like....

1 - Maintain a list of any pointers used by the perform routine that could not be freed immediately, each with a not_safe flag set to 1.
2 - At the end of your perform routine examine the list and set any current members not_safe flags to 0 (the perform routine will use the new pointers for these next time round)
3 - setup a clock that is called in the new routine and calls itself every n milliseconds - in your clock routine check your list of pointers and if they are marked as safe free them, otherwise leave them there.

Crashing probably due to memory limits