Basic question: smooth envelope follower

I have an incoming audio signal like a voice or an instrument and want to let it control external effects, finally by sending out MIDI from Max. But I don't only need the signal strength, I want to smoothen the signal and apply compression, expansion, delay and so on. The result should look as if you would follow a singing voice with a fader.

I think what I try to explain is called a controllable envelope follower. But not for single waves, rather for longer periods of an audio signal, let's say up to one second.

Currently I am struggling with the very first step, smoothing the signal. I tried [average~], [degrade~], [sah~] and [adsr~] in several combinations but all of them were still to sensitive. Maybe a compressor could do the trick. Using the output of the levelmeter would also be an option but attack/release is not enough control.

I hope this explanation was clear. So far I've learned yet I should first tame the signal, then convert to numbers and process the rest afterwards. Although, I do not like mathematical orgies. Is there another approach to extract data from an audio signal in a less sensitive way?