Note-off in poly~

I'm making a sampler with poly, triggering groove objects with a midi-controller form outside the poly.

When something gets changed on my controller (a note gets triggered for instance) poly picks a new voice to do its thing. But how can I get poly remembering on what voice he did what?

The problem is that a note-off won't be sent to the right voice, it will be send to a new voice, so somehow poly should remember where the note-on of that key(from the controller) went to send the note-off of that key to the same voice.

It should be possible to hold one note, play others while holding it and then stop the note you were holding. So somehow poly should remember on wich voice the note is triggered so it can send it's note-off to that voice, now the note-off message would be sent to another voice.