You have lower baseline latency with good drivers, so the processing you do just adds to that baseline, and the amount of it you can do also depends on stability of drivers.
In my opinion and experience M-Audio cards are not really good...
10ms is as much as you can hope for in this domain though. But 10ms is a lot.
It all boils down to Samples/Buffer. I can drive my RME 32samples/buffer via Firewire with no dropouts at all.
I could get my Echo down to 64samples/buffer but was not stable, Apogee also.
Mackie, Presonus, M-Audio; dare to dream going lower than 128...
Smaller buffer, higher sampling rate => lower latency.
RME have their own USB and FW drivers, while Apogee attaches to Apples generic Audio FW implementation.