Any way to clear jit.matrixset without audio glitching?

I've hit a problem in a large maxforlive patch which uses jitter.

I have a series of video buffers contained in jit.matrixset objects.
4 of these to be precise: jit.matrixset 200 4 char 640 480

If I send a 'clear' message to one of the matrixsets then ableton audio pauses.
I do have alot going on in the patch and may have hit some kind of limit somewhere but weirdly if I import a movie into the matrixset the audio doesn't glitch!

I have tried putting deferlow infront of the clear message (random act!) but it made no difference.

Do you think there is a way around this apart from using Max instead of M4L?

I have 8GB of Ram on this machine (running OSX 10.6).