This may seem a silly question but if my suspicions are correct then I don't have a problem.
As buffer~ just loads into RAM, presumably if an MP3 is imported then the buffer is cleared (i.e. all set to zero), then that memory is freed?
Example:
I import an MP3 into buffer.
I clear buffer using clear message.
I import another MP3 into buffer.
The memory which was taken up by the first uncompressed MP3 is now replaced with the new MP3, no loss of memory?
If this is the case then no worries. I have 2GB or RAM. I think that'll be sufficient.