How to Scrub through longer samples with Groove?

Andy Fennesz's icon

With samples up to 4 or 5 seconds in length, I've been able to use a Flonum attached to the groove inlet to change the sample start time, thereby scrubbing through the sound while it's playing.

I've since loaded a 20 second sample, though, and varying the Flonum value just produces silence.

Am I trying to do something impossible with long samples? Thanks.