Tristan Jehan’s analyzer~ object has a brightness and noisiness outputs that might help you identify some of these spots, and ignore them. You might as well use that as your pitch tracker, too. Or, re-write sigumund to give you some of that data.
Since a lot of the noise and consonants are in much higher frequency ranges than vowels, the most simple solution is maybe a corresponding Highpass filter.
You could gate the incoming sound, so it gets cut if the highpassed signal has a certain amplitude.