It was a bit temperamental, but this seems to be working:
The best way to play groove~ is with a signal. You can also use line~ if you want smooth adjustments. I'm not sure what you plan to do with this next, but definitely look at info~ if you need to do any sample selection. There is a good example in the polybuffer~ help file.