poly~ fixed to dynamic

i have this huge poly~ in a sampler with a high number of parameters per instance. i use pattr objects to access the intances from outside.
i already successfully optimized cpu usage with another nested poly~ for optional lfo and envelope instances and usage of mute to thispoly.
but it's still to heavy (ram) because the fixed number of instances (128). it needs to be maximum 16 or 32 times polyphonic.
A dynamic system would reqiure to send all the parameters per sample to the free instance before the noteon is passed.
My question is how to do it without adding latency for the sample playback. i'm serious, 1ms more is already to much i'm a realtime fanatic. How fast is pattr? so instead of a huge list from a coll i could send a number and use bindto messages to pattr objects. How do you handled dynamic poly~s with many parameters per voice? O.