Long term changes in audio roundtrip latency

1offby's icon
Max Patch
Copy patch and select New From Clipboard in Max.

Max 7, OS X 10.10.2
Focusrite Saffire Pro 40 FireWire interface
I have been noticing that, in a patch that takes audio in, processes it, and sends the audio back out, latency can increase dramatically when the patcher is running for multiple days.
Not sure this is entirely true, that MaxMSP is culpable, or what, but this is my current area of troubleshooting.
My results for a 15 hour test seem to show roundtrip latency tending to increase. Can someone explain this? In real-world use I have found that SOMETIMES, when the patcher I am using is on for days, the latency becomes extreme--slapback, echo, and beyond.
Maybe it's to do with the audio interface or its drivers? FireWire protocol? A shorter test using Macbook Pro builtin audio I/O was much more stable.
The test measures time in samples between click~ output to a speaker connected to the Focusrite interface via dac~ and onset detection of that click~ via an adc~ receiving input from a microphone connected to the Focusrite interface.
I just ran a 15 hour test using the pasted patch (curious whether this patch is even an accurate or good test) and got the following results:
Audio status: Sample Rate 44,1000, I/O VS 512, Sigvs 512, Overdrive on, Mixer Parallel Processing enabled.
mean time in samples:
hour 1: 1406
hour 2:1691
hour 3: 1659
hour 4: 1680
hour 5: 1601
hour 6:1795
hour 7: 1936
hour 8: 1770
hour 9: 1883
hour 10: 2124
hour 11: 2189
hour 12: 1990
hour 13: 1940
hour 14: 1861
hour 15: 1781
...maybe this would return to a mean but, anecdotally, I think it might continue to rise.

Graham Wakefield's icon
Max Patch
Copy patch and select New From Clipboard in Max.

The way that patch measures latency jumps between message and audio signal domains a few times (metro -> click~, adc~ -> edge~, bang -> count~ -> snapshot~). There could be variation here between the scheduler processing and the audio sample clock. It's better to measure latency purely in the audio signal domain. Here's a purely signal-domain measurement -- does this also show gradually increasing latency?

1offby's icon

Thank you!
I definitely was not keeping enough stuff in the audio domain.
Trying yours now. I don't think I can do a 15 hour test today, we'll see what happens...

1offby's icon

Well, I might try harder later but looks like the roundtrip time is rock steady when the test takes place in the audio domain.
I'm gonna investigate scheduler slop and where I might be introducing some sketchy audio/scheduler interactions.
Very helpful test Graham, thanks again.
For the record I started looking into latency bloat because last time the issue occurred I was able to document roundtrip times an order of magnitude longer than expected.

1offby's icon

Following up.
I just left the system on for a couple days and the unexpected latency increase is back.
I was not running Graham's patch the whole time so I cannot determine hour-over-hour change.
Using Graham's patch I am now seeing roundtrip latency of ~14,500 samples/30+ MS with the following settings:
SR:44.1 IOVS 512 SIGVS 512 Overdrive on Parallel processing on.
Disabling/enabling Overdrive, SIAI, or Parallel processing does not change the roundtrip latency.
Quitting and restarting Max immediately resolved the latency, with roundtrip latency of ~1650 samples (seems about right for 2xvector+speed of sound from speaker to mic).
The always-on-patch uses jitter and MSP, and a lot of audio threads. Not sure how to narrow down whether this is to do with something I have done in the patching. But closing the always-on patch does not solve the latency; Max has to be quit. Does this indicate some kind of "memory leak" or other cumulative thing?
Any tips on how to troubleshoot this?

1offby's icon

Bumping due to additional info below. Quick recap: the issue is very long (500 MS+) latencies developing when MSP audio is on for multiple days.
OS X 10.10.2, Max 7.0.3
I’ve had some time to further troubleshoot this issue and wonder if the following shed any light on the issue.
I decided to compare I/O latency through vst~ when one instance of vst~ is hosting Apple’s AUMatrixReverb and another is hosting a Lexicon PCM reverb.
To do this I modified Graham Wakefield’s Gen~ routine to compare sample delays of
1) a click~ passed through Gen~ with a single sample delay
2) a click~ passed through the AuMatrix Reverb then Gen~ with a single sample delay
3) a click~ passed through the Lexicon then Gen~ with a single sample delay
The plugins were set to 100% dry, 0 pre delay, 0.0dB gain.

Initial results showed the AuMatrixReverb having the same single sample delay as the unprocessed click~, and the Lexicon having a consistent 33 sample delay.
(sending the vst~ objects a bypass 1 message reduced the Lexicon to a 1 sample delay.)

This seemed unremarkable—Lexicon’s 33 sample delay is probably some internal buffer/whatever.

The patch sampled this delay time every second. Based on the experiences I’ve had I expected:
A) the Lexicon sample delay to increase either suddenly or hour-over-hour
B) the AuMatrixReverb and unprocessed click~ to remain stable

Instead, after running 24 hours all three sample delays were totally consistent.
But
OS X popped up a warning about Max.app exceeding memory limits. Upon investigation Activity Monitor showed Max using 64.44 GB Memory. (See MaxBigRAM.png) Max froze.
I decided that this may be due to using jit.cellblock to monitor the recorded sample delays, and asking jit.cellblock to refer to a too-large coll.
So
I removed it.cellblock and ran the patch again. After 14 hours Activity Monitor showed 39.69 GB Memory (See MaxBigRAM02.png)

Is this an issue related to Max? If this is some kind of "memory leak" how can I track down the source?

{
    "version" : "Version 7.0.3 (37355d2) (64-bit mac)",
    "platform" : "mac",
    "arch" : "x64",
    "osversion" : "Mac OS X Version 10.10.2 x86_64",
    "samplerate" : 44100,
    "iovs" : 128,
    "sigvs" : 128,
    "scheduler_in_audio_interrupt" : "off",
    "audio_drivername" : "Core Audio",
    "audio_driver_subname" : "",
    "license" : "permanent full",
    "machine_id" : " hi ”,
    "eventinterval" : 1,
    "overdrive" : "on",
    "mixerparallel" : "on",
    "mixercrossfade" : 0,
    "mixerlatency" : 30.0,
    "mixerramptime" : 10.0,
    "videoengine" : "avf",
}

MaxBigRAM.png
png
MaxBigRAM02.png
png
NewLongTermLoop02.maxpat
Max Patch
1offby's icon

FYI when I close the patch Max memory use remains high..this is the Activity Monitor after all patcher windows are closed.

RAMfullpatchclosed.png
png
Graham Wakefield's icon

If there's a memory leak, we need to eliminate all other possibilities. Could you run the test again, this time without any VST or AU plugins loaded, and without the coll writing, i.e. with the following patch?

Max Patch
Copy patch and select New From Clipboard in Max.

(Because there's a fair chance that the memory usage is from the plugins themselves, or from the coll sorting/writing, etc.)

1offby's icon

Graham, thanks for looking into this.
FYI I was just starting a process-of-elimination test, taking out one object at a time, first running a test without the Lexicon plug, two AUMatrixReverbs instead, and seeing the app memory steadily rise.
So, I'd suggest:
--The issue is not the Lexicon plug--
But sure, let's work the other way, from as few objects up. Process-of-nonlimination?
I will quit that and try the patch attached above. I wonder if you've tried the patch you just sent?

1offby's icon
Max Patch
Copy patch and select New From Clipboard in Max.

Hello Graham (or anybody), in a couple hours I'll be checking the memory usage of the patch Graham posted April 29.
I have a hunch it will be stable at about
Next I am thinking of trying this, with one instance of vst~. Does that seem reasonable:

1offby's icon

OK, Patch posted by Graham April 29 shows no increase in Max memory usage, sticks at 122MB.
Trying the one posted by me above this post.
Still doesn't have coll....

1offby's icon

Well,
The latest troubleshooting steps on this thread have only exposed a memory issue with coll:
https://cycling74.com/forums/writeagan-to-coll-memory-impact/
Nonetheless there really is a condition that causes MSP audio latency to grow longer over time, and I will continue to track it down.

1offby's icon

Back again with a thrilling update.
This morning the latency was back, after running 5 days. Using the measurement patch pasted below audio roundtrip averaged 14,400 samples@44.1SR=326 MS
Quitting and restarting Max audio roundtrip returned to 1,741 samples@44.1SR=39MS
Ideally this patch would be "always on." Any tips about where to look for the issue?
MSP-wise there's delay~ cascade~ vst~ doing lots of stuff.
This is a tricky because the latency only crops up after multiple days worth of uptime.

{
    "version" : "Version 7.0.3 (64-bit mac)",
    "platform" : "mac",
    "arch" : "x64",
    "osversion" : "Mac OS X Version 10.10.2 x86_64",
    "samplerate" : 44100,
    "iovs" : 512,
    "sigvs" : 512,
    "scheduler_in_audio_interrupt" : "off",
    "audio_drivername" : "Core Audio",
    "audio_driver_subname" : "",
    "license" : "permanent full",
    "machine_id" : "koala",
    "eventinterval" : 1,
    "overdrive" : "on",
    "mixerparallel" : "on",
    "mixercrossfade" : 0,
    "mixerlatency" : 30.0,
    "mixerramptime" : 10.0,
    "videoengine" : "avf",

1offby's icon
Max Patch
Copy patch and select New From Clipboard in Max.

The latest test patch:

Graham Wakefield's icon

Hi -- I've raised this internally to see if there are any other ideas, since running tests over 5 days is rather impractical! (And that got memory-use of coll writeagain back down to reasonable levels.)

Q: are you still using the Focusrite Saffire Pro 40 FireWire interface?
Q: when the latency went up, did you happen to see if there was an increase in memory usage?

1offby's icon

Here's another clue:
The issue persists after the patch is closed. Max needs to be restarted to resolve it.
UNLESS I go to audio status>change the input and sources from the aggregate device to Built In input & output and back.
Then the latency disappears.

Thomas's icon

I have just come across this thread and my current issue that I posted here seems very much the same.
(see: https://cycling74.com/forums/increase-in-audio-latency-over-time-pianointerfacemaxinterfaceheadphones/)

One (maybe very useful) difference is the speed with which the latency builds up. I haven't done proper testing yet, but it feels like an increase of at least 100 ms in 60 minutes.
Another difference is that there is nothing happening to the audio signal in Max. I am using Max just to reroute dynamically certain inputs to certain outputs (patch included in the other post).

I will carefully go through this thread now and I will also try to develop some audio-only latency test and report them back here and in the other thread.

1offby's icon
Max Patch
Copy patch and select New From Clipboard in Max.

Thomas,
Thank you for posting here. Since my last post on the thread, and with some help from Andrew Pask and David Zicarelli, I have concluded that the issue I was experiencing was not within Max or due to something in MSP but was instead part of the OSX audio I/O.
The early posts in the thread, and my early tests, revealed bogus latency issues due to poor patching or other stuff, such as a memory leak in coll (since resolved).
The results you are reporting--100MS over 60 minutes--are much greater than what I was finding.
My solution, pasted below, uses an automatic routine to disable MSP audio, disable and re-enable the MSP audio driver (in the pasted example note the audio driver is called "Saffire"), and re-enable audio once a day. This has allowed the patch to run without any latency buildup for multiple weeks.
I think what you are experiencing is different. What operating system/Max version are you using? is "Mixer Parallel Processing" enabled? What about "Mixer Crossfade?" What are the settings in the Max "Audio Status" window?

Thomas's icon

Thank you for responding!

I am still not sure how to test the latency. It seems now that I want to test it, it is actually not occurring.

The issue might as well be due to my poor patching as I still consider myself a self thought amateur and was building a quite complex (at least for me) patch. However, the audio signal is not really altered/part of the complexity, just routed (patch included in the end).

Unfortunately, I can't do any more tests today, but I will report back tomorrow.

Mac OS X El Capitan, Max 7.0.6 (32bit)
Mixer Parallel Processing is not enabled
Mixer Crossfade is off

Audio Status:
Driver: Core Audio
Input: Zen Studio
Output: Zen Studio
Sampling Rate: 44100
I/O Vector Size: 128
Signal Vector Size: 128
Scheduler in Overdrive: yes Audio interrupt: yes
CPU Limit: 0

Max Patch
Copy patch and select New From Clipboard in Max.

Thomas's icon

Great.. now that I finally set up a roundtrip latency test the latency issue disappeared and was replaced by an increase in crackling over time.
Everything starts out smoothly and after about an hour there is hardly any pitch left when hitting a piano key, just crackling.

1offby, when you use the Saffire, I guess your Input and your Output in the Max Audio Settings is set to Saffire, but what do you select as a driver?

1offby's icon

>>1offby, when you use the Saffire, I guess your Input and your Output in the Max Audio Settings is set to Saffire, but what do you select as a driver?<<
The audio driver I use in this case is: Core Audio
Since this issue is no longer about latency, I suggest searching for a related thread or starting a new one.
Some things to try if they have not already been tried:
Disable Audio Interrupt
Enable Parallel Processing
Increase buffer size (if the machine being used is relatively new 128 samples should be sufficient, nonetheless worth testing. FYI the MSP defaults are I/O Vector 512, Signal Vector 64).
Looking at the patch you posted Nov 18, I see 2 channels from adc~ being summed on input to matrix~, then split again on output from matrix~. Note that each signal inlet or outlet in MSP represents a single channel rather than a stereo pair.

Massimiliano Cerioni's icon

Hi everyone, I opened a new thread to share an alternative gen version of this, using the old cable loop trick with two buffers to record the click. Please tell me what you think!

https://cycling74.com/forums/a-new-io-latency-test-patch-made-in-gen