Audio Workflow for VR (max worldmaking package)

Yep if you're pulling the devel branch of the VR package you'll see I've been exploring some spatial audio too, but bear in mind this is all work in progress and is probably a bit of a mess. Once it's working well and cleaned up I'll shift it to the master branch & update the version in the package manager. For the adventurous I'll describe what I've done & where I'm hoping to go -- hoping someone might want to pitch in!

There are so many options out there, but mainly I've been seeing if I can get an object working for direct path HRTF and doing as much of the spatialization in MSP/gen~ as possible. Partly this is because I think Max's strengths are in the freedom to experiment with different ideas. However I think it's also vital to have something there in the VR package that 'just works' for demonstration purposes, which means also platform- and license-friendly. E.g. Oculus's VST plugins are unfortunately 64-bit only and I'm not sure about the license, but more importantly it's all packaged up in a plugin that means you can't get too experimental with it. I've also looked into the Oculus Audio SDK, though I can't go much further until they make it public. Would be great to look into Google's Resonance too, but that also doesn't have a C/C++ SDK yet. Of course that could just be the shape of my ears. I'll also look at spat~ soon, though that's only going to be useful to forumnet members.

I also tried plugging into the HOAlib externals, but I found the HRTF to be not as spatially clear. The idea of rendering all sources to an ambisonic soundfield and then panning that is appealing in principle, but in everything I've tried so far, the result is spatially muddier than direct HRTF encoding of each source. It's not difficult to make each source relative to head pose before encoding, rather than rotating a sound field, and for some of the things I've tried the CPU cost of the HRTF didn't seem that high, so I'm focusing on that for now. An ambisonic->HRTF path could be added later for larger numbers of sound sources.

Right now the most progress I've had is via the Steam Audio API. Although it's not perfect by any means, and there's been a few bugs (most of them ironed out with the last release), the library is in active development, covers all platforms, and has a suitable license. There's also some room simulation stuff there that would be great to get into later. The direct HRTF encoding has a little bit of noticeable shifting between HRTF slices that I think could be eliminated, but I think it originates from deeper in the Steam Audio library and I haven't been able to get rid of it yet. I also found that the Steam library was not encoding direction per ear (this makes a quite perceivable difference when sound sources are closer than about a meter), so I added an option to compute distance and angle and run a separate HRTF for each ear. Short range sound sources are appealing for VR usage, and it definitely sounds better with separate ear paths.

Other than that I've been exploring direct distance cues (amplitude, air absorption filtering, Doppler delay, reverb mix) via gen~. This should be independent of what HRTF library is being used anyway. So far so good. The only stumble I'm having is in interpolation of distance for the sake of Doppler, which I'll describe below in the hope that someone has a good idea or motivation to help! Basically, since we're receiving head-tracking updates at approximately 90 Hz, we need to somehow interpolate/approximate intermediate poses to create a smooth distance curve for each sample at 44.1kHz or 48kHz. A linear interpolation is passable for amplitude attenuation, but unusable for Doppler -- the sharp transients at each linear segment result in a step-like pitch shift of the delayed signal (like a robotic modulation) which is unusuable. The ideal would be to come up with an interpolating filter that produces a very smooth output (minimal 2nd derivative, minimal ripple). It doesn't even have to exactly hit the data points being received, since it's less perceptually important that distance is accurate (and anyway tracking data has temporal jitter), than it is important that the changes are continuously smooth. I thought a Kalman filter might fit the task but I honestly can't figure out how to write it. I've been messing with another filter design that is similarly based on prediction & gradual error correction, and seems to do OK mostly, but it still blows up sometimes. So far I've been doing the filtering on the raw distance data, but I guess there might be an advantage to filtering the object and head pose data instead, since we might expect certain consistencies to object and human trajectories. Anyway if anyone has interest & ideas in this regard, they'd be very welcome!