memory leak in max/jitter ?
Hi All,
i wrote a simple webcam app to shoot a long duration (one year)
time-lapse movie;
i use an imagingsource 1024x768 firewire camera
(http://www.1394imaging.com/en/products/cameras/firewire_color/dfk31af03/overview/
) which works perfectly well in jitter through iidc commands;
The patch uses just some basic objects (jit.qt.grab, jit.qt.record,
jit.lcd,...) and some simple patched logics to capture one image
every 30 sec during daylight, save one quicktime movie every day of
all grabbed frames in uncompressed format, and execute during the
night some unix scripts (through shell object) to prepare and ftp all
files to some server...
but the application systematically crashes after 5 or 6 days...
indeed process virtual memory size is expanding dramatically with
time... from 700Mb at the beginning, it doubles 24 hours later...
ah yes, and i also use 'mxj now' to get date and time every second
and to control start and end time of grabbing according to some
calendar data.
so?
on a macmini g4 OSX 10.4.7, max 4.5.7 runtime, jitter 1.5.2
--
Yves Bernard yb@imal.org
asbl iMAL vzw
rue d'Alost straat 7
1000 Bruxelles/Brussel
tel 32 2 213 37 10
BUG REPORTING GUIDELINES
Please report any problems you experience with clear and complete
information, including steps to reproduce, software and system
information, and where possible, an isolated example patch and crash
log. Something like the following would be ideal. This makes it
easier for us to find and fix the problems you experience. Without
such clear and complete information, it is less likely we will be
able to.
Summary:
Provide a descriptive summary of the issue.
Steps to Reproduce:
In numbered format, detail the exact steps taken to produce the bug.
Expected Results:
Describe what you expected to happen when you executed the steps above.
Actual Results:
Please explain what actually occurred when steps above are executed.
Regression:
Describe circumstances where the problem occurs or does not occur,
such as software versions and/or hardware configurations.
Notes:
Provide additional information, such as references to related
problems, workarounds and relevant attachments.
Am 20.09.2006 um 14:02 schrieb Yves Bernard:
> so?
Hi Jeremy,
i think my previous mail described most of the points you require but
an isolated example patch with crash log;
i will try to do an isolated example patch (currently i am flying
away for a week), but in the meanwhile, maybe there are some known
bugs or well known things which provoke memory leaks? Maybe some
memory issues have been solved in 4.6 ?
here i reformulate my mail data according to your reporting guidelines:
>
>software and system information
on a macmini g4, 512Mb, OSX 10.4.7, max 4.5.7 runtime, jitter 1.5.2
>Summary:
the application systematically crashes after 5 or 6 days...
indeed process virtual memory size is expanding dramatically with
time... from 700Mb at the beginning, it doubles 24 hours later...
The patch uses just some basic objects (jit.qt.grab, jit.qt.record,
jit.lcd,...) and some simple patched logics to capture from a
firewire camera one image every 30 sec during daylight, save one
quicktime movie every day of all grabbed frames in uncompressed
format, and execute during the night some unix scripts (through shell
object) to compress and ftp all files to some server...
The patch uses 'mxj now' to get date and time every second and to
control start and end time of grabbing according to some calendar
data.
>Steps to Reproduce:
it is a non interactive application; it just grabs frames, adds them
to a daily jit.qt.recorded file, saves the file at end of the day,
starts again next morning; you just launch the application, let it
run and it will crash after some days...
>Expected Results:
NA
best
--
Yves Bernard yb@imal.org
asbl iMAL vzw
rue d'Alost straat 7
1000 Bruxelles/Brussel
tel 32 2 213 37 10
hi yves,
no solution here, but your description reminded me the notorious ram-
eating-sfplay discussion which last peaked in june. after blaming
max, it turned out, that there were some bad things going on with the
os file-caching.
i was bitten by that on exactly a mac mini with your ram size and
configuration, whereas my powerbook with more ram didnt show the
problem.
at that time my only solution was to watch the vram-size via shell-
object and, whenever it got below a certain value, script an
appropriately large buffer and immediately destroy it. that was the
only way to free the vram again (besides restarting the whole machine).
maybe there is more than just a coincidence in that?
hans
hans w. koch
im krahnenhof 11
d-50668 koeln
+49-221-554902
www.hans-w-koch.net
Am 20.09.2006 um 16:53 schrieb Yves Bernard:
> Hi Jeremy,
>
> i think my previous mail described most of the points you require
> but an isolated example patch with crash log;
> i will try to do an isolated example patch (currently i am flying
> away for a week), but in the meanwhile, maybe there are some known
> bugs or well known things which provoke memory leaks? Maybe some
> memory issues have been solved in 4.6 ?
>
> here i reformulate my mail data according to your reporting
> guidelines:
>
>>
>> software and system information
> on a macmini g4, 512Mb, OSX 10.4.7, max 4.5.7 runtime, jitter 1.5.2
>
>
>> Summary:
> the application systematically crashes after 5 or 6 days...
> indeed process virtual memory size is expanding dramatically with
> time... from 700Mb at the beginning, it doubles 24 hours later...
> The patch uses just some basic objects (jit.qt.grab, jit.qt.record,
> jit.lcd,...) and some simple patched logics to capture from a
> firewire camera one image every 30 sec during daylight, save one
> quicktime movie every day of all grabbed frames in uncompressed
> format, and execute during the night some unix scripts (through
> shell object) to compress and ftp all files to some server...
> The patch uses 'mxj now' to get date and time every second and to
> control start and end time of grabbing according to some calendar
> data.
>
>> Steps to Reproduce:
> it is a non interactive application; it just grabs frames, adds
> them to a daily jit.qt.recorded file, saves the file at end of the
> day, starts again next morning; you just launch the application,
> let it run and it will crash after some days...
>
>> Expected Results:
> NA
>
>
>
> best
>
>
>
>
> --
> Yves Bernard yb@imal.org
> asbl iMAL vzw
> rue d'Alost straat 7
> 1000 Bruxelles/Brussel
> tel 32 2 213 37 10
>
> http://www.imal.org
> http://www.i-cult.be
Am 20.09.2006 um 16:53 schrieb Yves Bernard:
> Hi Jeremy,
>
> i think my previous mail described most of the points you require
> but an isolated example patch with crash log;
Sorry, but this has come up countless times before. Without the
patch, your description of "some basic objects" and "simple patched
logic" could mean anything. If the question is, "are there any known,
significant memory leaks in Jitter?" then the answer is "not that
we're aware of". But there's no chance that this is going to be
looked at without a patch that can show it happening (without any 3rd
party objects, please).
You might also try out the 1.5.3 public beta to see if something got
inadvertently changed/fixed along the way.
> The patch uses just some basic objects (jit.qt.grab, jit.qt.record,
> jit.lcd,...) and some simple patched logics to capture from a
> firewire camera one image every 30 sec during daylight, save one
> quicktime movie every day of all grabbed frames in uncompressed
> format, and execute during the night some unix scripts (through
> shell object) to compress and ftp all files to some server...
On Sep 20, 2006, at 7:53 AM, Yves Bernard wrote:
> maybe there are some known bugs or well known things which provoke
> memory leaks? Maybe some memory issues have been solved in 4.6 ?
The Apple File Manager caching sfplay~/sfrecord~ issue which Hans
mentioned has been fixed in 4.6. However, if Quicktime itself uses
standard file calls internally, rather than the calls which
explicitly disable Apple File Manager caching, then this general
problem with Apple's File Manager caching algorithm may demonstrate
similar behavior. If this is the case (for which I have no evidence
to assume it is), as far as I know, there's nothing we can do. It
appears to be an Apple issue. I've mentioned this curious behavior
(which is only exhibited on certain machines for certain people,
according to some mysterious sequence of events) to some people at
Apple, and perhaps it will be addressed in a future release.
As Jeremy mentioned, without an example patch to demonstrate clearly
what you are doing, however, there's really not much else we can do
to help you. From our standpoint, as developers interested in
providing assistance and fixing bugs where possible, reports without
clear examples are extremely frustrating (and typically viewed as an
impolite drain on the mailing list...more info on this in the oft
cited "How To Ask Questions the Smart Way").
> the application systematically crashes after 5 or 6 days...
> indeed process virtual memory size is expanding dramatically with
> time... from 700Mb at the beginning, it doubles 24 hours later...
> The patch uses just some basic objects (jit.qt.grab, jit.qt.record,
> jit.lcd,...) and some simple patched logics to capture from a
> firewire camera one image every 30 sec during daylight, save one
> quicktime movie every day of all grabbed frames in uncompressed
> format, and execute during the night some unix scripts (through
> shell object) to compress and ftp all files to some server...
> The patch uses 'mxj now' to get date and time every second and to
> control start and end time of grabbing according to some calendar
> data.
If you are generating new symbols based on the time/date, and/or
generating symbols with the shell object, you might be growing the
symbol table without end. Keep in mind that symbols have infinite
lifetime in Max, and there's no way to flush the symbol table. So if
this is your issue, you'll need to figure out a way to either change
the way your patch is implemented to be more memory efficient, or
automatically restart your patch.
-Joshua
here is a stripped version wich just shows the constant memory
increase of the max runtime process (Vsize as shown by unix 'top' or
Activity Monitor);
after 155 grabed frames (about 75 minutes), the process size has
increased from 618Mb to 700Mb...
a lowres jpeg preview of each grabbed frame is uploaded by ftp to
some server; this is done through a shell script (uploadJpg.sh); its
code is given as a comment in a subpatch; save the text of this
comment in a unix -x file;
you have to edit some message boxes to set up your ftp account,
various path and file names;
then you have to edit the coll in 'myDate' subpatch to modify the
grabing periods of the weekdays;
and of course, make a runtime including all required externals
(jitter, java for mxj);
the only external is Bill Orcut's shell object.
best,
>On Sep 20, 2006, at 7:53 AM, Yves Bernard wrote:
>
>>maybe there are some known bugs or well known things which provoke
>>memory leaks? Maybe some memory issues have been solved in 4.6 ?
>
>The Apple File Manager caching sfplay~/sfrecord~ issue which Hans
>mentioned has been fixed in 4.6. However, if Quicktime itself uses
>standard file calls internally, rather than the calls which
>explicitly disable Apple File Manager caching, then this general
>problem with Apple's File Manager caching algorithm may demonstrate
>similar behavior. If this is the case (for which I have no evidence
>to assume it is), as far as I know, there's nothing we can do. It
>appears to be an Apple issue. I've mentioned this curious behavior
>(which is only exhibited on certain machines for certain people,
>according to some mysterious sequence of events) to some people at
>Apple, and perhaps it will be addressed in a future release.
>
>As Jeremy mentioned, without an example patch to demonstrate clearly
>what you are doing, however, there's really not much else we can do
>to help you. From our standpoint, as developers interested in
>providing assistance and fixing bugs where possible, reports without
>clear examples are extremely frustrating (and typically viewed as an
>impolite drain on the mailing list...more info on this in the oft
>cited "How To Ask Questions the Smart Way").
>
>>the application systematically crashes after 5 or 6 days...
>>indeed process virtual memory size is expanding dramatically with
>>time... from 700Mb at the beginning, it doubles 24 hours later...
>>The patch uses just some basic objects (jit.qt.grab, jit.qt.record,
>>jit.lcd,...) and some simple patched logics to capture from a
>>firewire camera one image every 30 sec during daylight, save one
>>quicktime movie every day of all grabbed frames in uncompressed
>>format, and execute during the night some unix scripts (through
>>shell object) to compress and ftp all files to some server...
>>The patch uses 'mxj now' to get date and time every second and to
>>control start and end time of grabbing according to some calendar
>>data.
>
>
>If you are generating new symbols based on the time/date, and/or
>generating symbols with the shell object, you might be growing the
>symbol table without end. Keep in mind that symbols have infinite
>lifetime in Max, and there's no way to flush the symbol table. So if
>this is your issue, you'll need to figure out a way to either change
>the way your patch is implemented to be more memory efficient, or
>automatically restart your patch.
>
>-Joshua
--
Yves Bernard yb@imal.org
asbl iMAL vzw
rue d'Alost straat 7
1000 Bruxelles/Brussel
tel 32 2 213 37 10
This is missing an object called "c2", as well.
I am intending to test this today. However, Yves, I would prefer if,
in the future, you would please take more care to limit example
patches to only the elements which cause the behavior. As it is, I
will need to test the grabbing section, the ftp/shell script section,
the mxj section and the preview jpeg section, all independently,
which will take some real amount of time.
A better approach, from my perspective as someone attempting to see
this problem (in order to potentially fix it), is for you to have
already figured out what part of your patch is not working properly
and simply show it to me. This is, I would argue, your responsibility
as a developer and as the reporter of a bug. And, if I have trouble
reproducing this bug, this is exactly what I'm going to ask you to do.
Please point me to "c2" and we'll take it from there.
Best
jb
Am 21.09.2006 um 00:20 schrieb Yves Bernard:
> here is a stripped version wich just shows the constant memory
> increase of the max runtime process (Vsize as shown by unix 'top'
> or Activity Monitor);
> after 155 grabed frames (about 75 minutes), the process size has
> increased from 618Mb to 700Mb...
>
> a lowres jpeg preview of each grabbed frame is uploaded by ftp to
> some server; this is done through a shell script (uploadJpg.sh);
> its code is given as a comment in a subpatch; save the text of this
> comment in a unix -x file;
>
> you have to edit some message boxes to set up your ftp account,
> various path and file names;
> then you have to edit the coll in 'myDate' subpatch to modify the
> grabing periods of the weekdays;
> and of course, make a runtime including all required externals
> (jitter, java for mxj);
>
> the only external is Bill Orcut's shell object.
>
>
> This is missing an object called "c2", as well.
Hi Jeremy,
oups, sorry, here is 'c2':
>
> I am intending to test this today. However, Yves, I would prefer if,
> in the future, you would please take more care to limit example
> patches to only the elements which cause the behavior. As it is, I
> will need to test the grabbing section, the ftp/shell script section,
> the mxj section and the preview jpeg section, all independently, which
> will take some real amount of time.
>
> A better approach, from my perspective as someone attempting to see
> this problem (in order to potentially fix it), is for you to have
> already figured out what part of your patch is not working properly
> and simply show it to me. This is, I would argue, your responsibility
> as a developer and as the reporter of a bug. And, if I have trouble
> reproducing this bug, this is exactly what I'm going to ask you to do.
>
the patch is working properly: it does perfectly what it he is designed
for... there is just this memory leak (and patch developers do not have
any way to control memory management which is the responsability of the
max memory system/garbage collector).
yes it takes time to try to see where the problem is...
this is why i send a first mail to the list to see if there were any
know issues with memory...
currently, i am away without any equipment to test it... but once at
home after 9/27 i will try to do a version with the exact
functionalities causing the memory leak
> Please point me to "c2" and we'll take it from there.
--
Yves Bernard
asbl iMAL
Centre Dansaert, 7 rue d'Alost
1000 Bruxelles
yb@imal.org
+32 478 27 55 10
>
>
> This is missing an object called "c2", as well.
Hi Jeremy,
and 'c2' uses 'strcat' from jasch (Jun 23 2006 version)...
--
Yves Bernard
asbl iMAL
Centre Dansaert, 7 rue d'Alost
1000 Bruxelles
yb@imal.org
+32 478 27 55 10
Thanks, I've found the problem, which should be resolved in an
upcoming release of Jitter.
jb
Am 22.09.2006 um 19:39 schrieb Yves Bernard:
>> This is missing an object called "c2", as well.
>
> Hi Jeremy,
Hi Jeremy,
i just come back from bucarest... and read your e-mail only now after
doing my own research back at home today;
from my analysis the memory leak is caused by 'exportimage' message
sent to jit.matrix or jit.qt.grab (maybe there is a thread problem
behind this as you suggest).
here is a patch which identifies the problem;
(you can take off the 'jit.matrix @adapt 1' and just send the
'exportimage' message directly to jit.qt.grab and it has the same
effect)
This problem is happening also with the latest version max 4.6.2/jitter1.6.2;
suggestion: while you are looking at export, could it be possible to
specify compression quality with numbers (e.g. from 0 to 100) instead
of a few discreet values (low, normal, high, best)?
>Thanks. I discovered the problem after spending lots of time with
>the patch. Turns out to be a complicated problem related to thread
>maintenance, but it will be fixed in the next Jitter release.
>
>Thanks,
>jb
>
>
>Am 23.09.2006 um 17:39 schrieb Yves Bernard:
>
>>i've tested this stub-jitter version for near one hour and a half
>>(about 150 'grabbed' frames), and the memory stays stable around
>>590Mb.
>>(orginal patch was increasing about 100Mb...)
--
Yves Bernard yb@imal.org
asbl iMAL vzw
rue d'Alost straat 7
1000 Bruxelles/Brussel
tel 32 2 213 37 10
http://www.imal.org
http://www.i-cult.be
jitter mailing list
jitter@cycling74.com
https://cycling74.com/mailman/listinfo/jitter