Having trouble with mc.poly~ / the Vocalese M4L plugin, Voices eventually fail.

Having mc.amxd~ Vocalese not work, I decided to just copy it and convert some parts to MC. Wanting to perform a simple 'deviate' on the sig~ that drives the groove~ inside the poly~. I got it to work as expected, but eventually all the voices of the poly~ start to fail or disappear.

I think I have proper note off messages, I tried busymap attribute, the voices are getting muted.

I'd really appreciate any direction anyone can give.
Honestly I'm not even sure how the mc.poly~ is receiving the information to play different sections of the buffer~, it seems to me its only getting 1 message from the coll .

attached is my multi channel patch (which is basically identical to the Vocalese plugin, a copied/renamed vocal.Voice~ poly and copied/renamed coll file.