Ballpark recommendations are usually for larger IO buffer and smaller sv, where there's no realtime audio input. Trial and error will let you nail specific sizes, dependent on OS and hardware setup.
Further replies will comment on the direct impact (perceptual and actual) of lower/larger sizes. Of which I know nada.