I don't know, i didn't have a very close look at your patch(tidying up helps you and us debug..)
Updating the frequency of the oscillator at audio rate (via [sig~]--[mtof~] --[sah~]) would be worth a try. What attack times are you using? Do you have clicks in the transients or attack phase or also at the end of and envelope? Adjusting retrigger time could also help, although the default values shouldn't click i think.
I have a dial that have an attack set up to 500 milliseconds. I have clicks at the end of an envelope and when I'm playing more than one note at a time. I tried to do something with the retrigger, but it didn't do anything. Not sure if I was understanding how to do it correctly.
Read the help files about what a target 0 message does to the [poly~ retrysynth 10 @steal 1], because each of the [adsr] wants to get its values, not only the one in the first instance. I Hope that helps.
poly~ reference about the target message: Specifies the poly~ instance that will receive subsequent messages (other than messages specifically used by the poly~ object itself) arriving at the poly~ object's inlets - for example, The message target 2 routes messages to the second instance. If the target message specifies a value greater than the current number of instances (copies) of the loaded patcher, the message will be sent to the highest numbered instance (e.g., sending the message target 2 to a poly~ object containing only a single instance will send subsequent messages to the first instance). The message target 0 sends input to all instances, and using any negative number value with the target message will disable input to all instances.
For sending messages to the [poly~] itself use its first inlet.