record~ loop in tape-style guitar looper: problems with overdubbing

Hey everyone! I am trying to create a stacking looper based on this Andrew Robinson's video with variable speed and the option to overdub while the groove~ object is playing at different speeds.

There are two buffers, one for recording and one for playing. The recording buffer gets copied in the playing one and the groove~ object that plays the playing buffer is fed back into the record~ object to be written into the recording buffer.

Here you first need to create a base loop with the top left toggle e then with the top right toggle you can overdub to that loop.

I have a doubt about the overdubbing: if I just record a base loop, switch off the top left toggle and then switch on the top right (overdub) toggle and just let it sit there with no input I notice that my waveform shifts to the left with different step sizes every time. I am not sure about what could cause this behaviour.

Has anyone tried to do something similar? Thank you in advance!