Clicking issue when changing grain size of granulator

I've been bashing my head in for days trying to figure out what I'm doing wrong... hopefully someone can help. I've made a very simple monophonic granulator patch in Gen, which takes in a single value to change the size (in ms) of the grain from a buffer.

The grain has a Gaussian amplitude envelope, so there are no pops or clicks when the size is static. But every time I click and drag a number box (connected to the gen~ patcher inlet) up and down to change the grain size, I get a ton of clicks. Dragging quickly in order to make larger changes in grain size, say from 10ms to 1000ms, doesn't generate many clicks because the amplitude envelope in the gen patch retriggers so often—every time the inlet value changes—that the signal remains near silent until I stop changing the value. However, changing from 10ms to 12ms, produces these clicks.

I'm assuming the reason this occurs is because of discontinuities in the signal that occur every time the grain size is changed before the amplitude envelope returns back to 0. I tried to account for that by implementing Miller Puckette's "switch and ramp" technique described here, but it doesn't seem like it's working. Or I'm misunderstanding something! Does anyone have any ideas on how to solve this issue?