Most likely you test this patch with an io vector size of 64 samples, and audio in interrupt turned on. Here you hit the limits of precision of the audio to non-audio domain communication. With the settings mention this happens on a multiple of 64 samples, always at the end of a vector size. Of course it would be possible to decrease the io vector size. On average however you will get 500 ms. The audio domain is precise.