can you track perspective of a chromakeyed thing?

Dante's icon

Is there a way to determine or estimate the perspective of a chromakeyed shape?

For example, I'm keying a green rectangle and want to replace it with a video or image, but I want the replacement to stay in perspective as the camera moves around slightly. I am looking into the cv.jit.perspective, cv getperspective, and findhomography, as well as cv.jit.shift. but I'm not sure I'm on the right path.

TestingPerspective.maxpat
Max Patch

Dante's icon

weekday bump: I tried using cv.jit.orientation to create the transformation matrix for getperspective. but I must not be loading the coordinates into the matrix properly, but even a rotation would not be perspective so idk.

orientationForum.maxpat
Max Patch