Is there a way to determine or estimate the perspective of a chromakeyed shape?

For example, I'm keying a green rectangle and want to replace it with a video or image, but I want the replacement to stay in perspective as the camera moves around slightly. I am looking into the cv.jit.perspective, cv getperspective, and findhomography, as well as cv.jit.shift. but I'm not sure I'm on the right path.