Poly~ hangup? Or something else?

I'v been struggling for the past three days with a synth that I'm building. It is pretty straight forward. Instead of a groove, I'm using the Elasticx~ object. I'v set up my poly~ with 16 instances.
The way it is supposed to work is so that I can turn elasticx~on and off. Using it either as time stretch or a regular pitch.

All goes well until i crank up the release, turn elasticx~on off a couple of times and then..... It seems that one (sometimes more) of the instances of my poly~ does not update. If i turn off the elasticx~ then one or two instances keeps it on and vice versa. It seem that when the poly~is using all the 16 instances or close to it, it kind of misses to update some of them. If you hit ONE key long enough with the release cranked up to 127 then you can hear that suddenly one instance (sometimes more) is in a different pitch. What could be the reason. I cant seem to figure it out.