Have you looked at the MSP tutorials? All sorts of DSP math is done there.
– Max is the main framework and is (largely for historical reasons) focused on event-handling and MIDI and general GPL stuff.
– MSP is the DSP extension, ranging from math primitives to ready-made filters and FFT and stuff.
– Jitter is the video/graphics extension (actually, a general purpose matrix manipulation environment)
– You really need to understand at least the basics of Max before you dive into MSP (or Jitter), or there will be tons of stuff that won’t make sense.