You are using the output of the adsr~ object to directly control the the cutoff frequency. While you hold a note the adsr~ object will stay in the sustain phase and output a constant value.
Try to make the adsr~ with an amplitude of 1 and multiply the output with the cutoff-frequency parameter. Like this:
P.S.: it should work, but i didn´t test it