Filtering a control signal without affecting the output.

Hello,

I am building a simple mono pan control and would like the control signal coming from the pan control to be filtered to avoid any pops or clicks when panning.
I have had no luck so far and anyway of filtering that I have tried has affected the main output signal.
The pan is built with two signal routes coming out from the left of the plug in with a multiply box on each, these then exit via the L/R of the plug out. The pan control signal is coming in on the other inlet of the multiply boxes.
Does anyone have a way to do this?

Thanks.