Buf.op on a long buffer

Ithamar Eshpar's icon

Hello all,

I've been trying to perform some simple Buf.op operations (gain, normalize etc) on some audio files, but it seems like there's a limit to the size of the buffer: when trying to operate on a 40-minute file, Max crashes with no warning..

It's strange because the file loads fine into the buffer, I can see it drawn in [waveform~] and the results of [info~] seem to be correct.

What's the maximum size of a buffer? Is there a workaround for working with long files?

I couldn't find any reference about it even though it seems quite basic information..

Thanks,

ith

Source Audio's icon

Did You try doing it without BufOp ?
You can send gain and normalise messages directly to buffer, a least since max 6.

Ithamar Eshpar's icon

Well, I've just begun experimenting with this, and my final goal is to modify the Buf.Op and add some more sophisticated operations, so using the built-in buffer operations won't be enough... And I do intend to work on large files.

Source Audio's icon

There is a object el.buffet~ by Eric Lyon that does buffer copy, fade in/out, normalise, gain, rotate zero etc
Unofficial version which was uploaded to forums few years ago had overdubb function included.
External is only 32 bit compatible, allows only 2 channels, selection only in ms but
it is really fast when executing buffer edits, and very reliable. BufOp being java is a snail compared to it.

Source code for version 3 /64 bit is available on github, so one can look into code, modify to own needs and compile it.
Cycling folks could have included much more of that stuff into standard buffer object capabilities.
Buffer copy was available in old days of Mac OS 9, bufcpy~by Bill Orcutt worked perfectly.

Ithamar Eshpar's icon

Ok, thanks for all that info, but I'm trying to build a very specific application which involves scanning a few buffers and manipulating them in several ways. What I need is not a ready external but some info that would help me build my own process.

So - is it impossible to process big buffers in MXJ Java? So far I didn't even succeed in scanning buffers that are longer than 2^24 samples (which is totally strange, as I understood that int numbers should go up to 2^32..) When my counter goes past 2^24 Max either hangs or crashes..

Emmanuel Jourdan's icon

The problem is likely to be that the sound file that you read is larger than the memory that the Java Virtual Machine has. You can increase the size of the JVM by changing the line in max.java.config.txt config file.

from max.jvm.option -Xmx256m to max.jvm.option -Xmx2048m

Ithamar Eshpar's icon

Thanks Emmanuel Jourdan!!

Changing to 2048 caused a "Can't create JVM" message, but then I tried 1024 and it did the trick! I guess I don't have enough memory for 2048 (I only have 4 GB on my computer).

One last question on this topic: I intend to build an executable application from my patch and send it to my clients - will it work on their computer or will they also need to perform some edits in order to make it run properly?

pdelges's icon

If you build a standalone using Java classes, you'll need to include a couple of files (Max won't do it for you), one of them is max.java.config.txt. So your clients won't need to edit anything.
You'll find more informations here: https://docs.cycling74.com/max7/tutorials/jitterchapter99_appendixd

Eric Lyon's icon

Thanks for mentioning el.buffet~. The object is part of LyonPotpourri 3.0, which may be downloaded at: http://disis.music.vt.edu/eric/main/lyonpotpourri/

el.buffet~ is a someone unusual signal object that does not call a perform routine. All its work is done NRT, which in practice is usually much faster than RT. The object should work just as well in 64-bit as 32-bit mode. Its computation is on 32-bit floats, but that is because Max audio buffers are currently 32-bit, not 64-bit. Also, the channel count is not hard-coded, and should scale up to whatever the channel count of the current buffer is. I have only tested up to 8-channel buffers, but there should be no problems for considerably larger buffer channel counts.

-Eric

Source Audio's icon

Unfortunately el.buffet in version 3 is having many problems,
not freeing buffer after operations, so that buffer remains locked - no
read, replace etc. and crashes way to much, in both max 6 & 7 versions.
I still use old one v2 which works perfectly.

Eric Lyon's icon

Please describe a series of steps to reproduce this problem, and I will investigate.

-Eric

Eric Lyon's icon

Using the el.buffet~ external from the distro listed above, I could not reproduce the reported problems. Tested on Max 7.3.4, 32-bit. With a more detailed bug report, I could investigate further.

-Eric

Source Audio's icon

Hi Mr. Lyon and thanks for the fast reply.
I am a happy el-buffet~user from the very beginning of it's existence,
even recompiled one version to accept samples as time units and work
on 4 ch buffers.
but the version 3 gives me troubles : normalize, fades, rotatezero, killdc and similar work without problem, but as soon as any copying takes place, buffer in use gets "locked", no reimportting samples, no crop etc.
Only errors "can't read fiIe now" and if one tries to crop ""can't resize now".
I tried to focus el.-buffet to some other buffer with setbuf message, thinking that buffer in use would get released, but nothing changed.
Then - stuff from the Reference : catbuf, events, pevents - don't get recognised as messages,
copy_to_mono_buffer does not copy anything.
Here is the screenshot of the errors:

overdub message crashes max 6.1.10 & 7.3.4, errors are
displayed very shortly as crash follows immediately.

Eric Lyon's icon

Thanks for the detailed report. I will check into these issues later in the month.

-Eric

Source Audio's icon

Any news ?

Eric Lyon's icon

Please give the attached build a try.

Thanks,

Eric

el.buffet~.mxo.zip
application/zip 32.50 KB

Source Audio's icon

Hi and thanks for all the efforts.
I can't make extensive tests at the moment,
only max 6.1.10 32 bit.
I will check it on max 7.3.5 32 and 64 bit as soon as possible.
From the first check - copy to buffer works ok.
Overdub crashes Max, message "method dirty called on invalid object"
gets posted in Max console
Update :
In Max 7.3.5 both 32 and 64 bit overdub also crashes max,
copy, fading, normalising, swap etc work fine.
copy_to_mono_buffer still does not copy anything.
One thing is different from original buffet version:
overdub message first wipes both source and destination
buffers, than crash happens

Source Audio's icon

And ... even that copying works now, buffer still gets locked - one can't crop it.
error buffet~: naughty start frame: 39670 ( could be any other position in buffer)
gets posted if one tried to crop buffer, after or before doing some buffet operations,
looks as if buffet does not get notice of buffer changes...
But I can't yet figure out what makes it happen exactly
Update - it seems that setchunk message sent when one selects
portion of waveform in help file produces the naughty start frame error.
It happens independent of any buffet operations, and only if
selection includes last sample of the buffer.

Eric Lyon's icon

Please try the attached. Note that el.buffet~ is a complicated Max 4 style external, which will require significant iteration for full modernization. Please be sure to report the specific message that causes each crash or unspecified behavior.

Thanks,

Eric

el.buffet~.mxo.zip
application/zip 17.26 KB

Source Audio's icon

Thanks !
I will give it a try and report back.

Source Audio's icon

This last version of buffet seems to be 64 bit only.
that's no problem as buffet version 2 works perfectly in 32 bit...
Good news is that overdub is working now.
If one starts max patch with set buffer lengths,
overdub works, but if one resizes the buffer prior to overdub function,
either by size $1 message or replace, max crashes as soon as overdub is executed.
On the other hand, if buffer gets resized by crop message, overdub works
and does not crash max.
After execution of overdub, destination buffer can be cleared,
croped, but resizing the buffer with size $1 message does not work.
If one crops the buffer, resizing works again.
Same applies to replace message to buffer.

To make it short, I have a feeling that buffet does not like when buffer
gets resized, but it copes with it if buffer gets croped...
--------------
Copy function does not suffer from the resizing problem as overdub,
but after executing copy function, source and destination buffers get "locked" ,
one can't execute crop, resize, read, replace , but max does not crash.
Errors can't resize now and can't read file now get posted to max console.
Overdub and clear functions do work.
--------------
copy_to_mono_buffer still does not copy anything, buffet bangs as if it did execute the copy.
--------------

error message buffet~: naughty start frame: is triggered when one sends mschunk message to buffet in case that end point is also buffer end.
Beginning of the selection gets reported in samples.
So if buffer is 1000 ms long , sending mschunk 200 1000 would produce error :
buffet~: naughty start frame: 8820

if endpoint is just 1 sample less than buffer size , than there is no error

Eric Lyon's icon

Thanks for the detailed report. See if any of those problems are fixed with the attached external.

-Eric

el.buffet~.mxo.zip
application/zip 19.00 KB


Source Audio's icon

Thanks, I will look into it tomorrow and report.
Cheers !

Source Audio's icon

Here is the report:

resizing buffer prior to overdub does not crash Max any more.
Great !!!!
One thing remains - if one has mismatch in buffer channels
which could happen if one by mistake replaces mono buffer
with stereo audio file and tries to overdub, max crashes without error.
Trying to copy from / to mismatched buffers does display error and Max does not crash.
--------------
copying still locks the buffers.
One can still overdub, clear , copy agin etc, but reading into buffer
or resizing it does not work, as before.
----------
info message to buffet crashes Max if no buffet operations were executed.
After any buffet action, info works.
error buffet~: naughty start frame is still there.
----------
I see that You also reactivated events and pevents functions, great !
----------
Another question - would You mind uploading updatet source of the buffet
either here, or to github.
The reason I ask is that I would like to modify buffet for personal use
to accept samples instead of ms, and add few functions like copy_from_buffer
or clearall ...
And again - thanks for looking into this, buffet is so valuable and
the only tool that makes looping functions like undo - redo
and many others possible.

Eric Lyon's icon

Here's the Github link:
https://github.com/ericlyon/LyonPotpourri3.0-MaxMSP

Also, another buffet~ build. I don't plan to add sample-level specification, since if the sample rate changes after you typed in your sample numbers, you're on your own. (Of course, what you do with the Github code privately is your own business.) See if sampstoms~ does what you need.

-Eric

el.buffet~.mxo.zip
application/zip 18.65 KB


Source Audio's icon

Thank You !
Just tried the buffet briefly, and copy seams to make no troubles any more.
Have to check all through though.
And I really appreciate that You allow to mod the code for personal use.