The main goodie here seems to be the control over the shape of what's projected, that it can focus upon only the pillars, everything else but the pillars, the top triangle, etc. This takes a bit of work but is totally doable with jitter using the GL objects, you could create the "scene" and have multiple shapes laying on top of a standard rectangular background, then have the objects on top mask the background, or have only the objects get the textures rather than the background, etc.
If you have any 3D modeling background you can save as .obj and bring in models to your jitter GL scene. So if you know what your projection area looks like (and what the physical objects you want to work with look like) you can make a bunch of models that match the layout, then apply/unapply textures to each as desired.
For more coolness, a camera which can determine where your objects are (based on their brightness probably) would be neat, then you could dynamically assign textures to certain areas and not to others. Turning up and down a threshold for detection could also be interesting, having the textures "bleed out" from the objects, or "collect into" them more precisely. Then it's all about capturing and projecting on the passerby... if they'll let you!
there are 3 ways of videomapping
-make a 3d model of the scene render it in a 3d app , project it on top of the architecture , like the one in the above movie
-make a photo with a camera mounted on top of the projector and create a mask
-use keystoning and warping to fit the architecture
I have done a project recently which uses similar features to map 3d objects with video and I think what you want to do will be pretty do able with some additional skills. Like seejayjames mentioned, you probably want to check out the jit.gl.object sutff. I would guess you will want to make replica of what you are mapping in 3d (maybe google sketchup if you don't do 3d studio max or maya) then map video as your texture which is easy.
All though you might be able to do similar stuff with masking and contrasting, I think the 3d method is the only way to get spatial effects like images wrapping around your object etc (but that might happen naturally and be ok)
I'll paste in some parts of the patch I was using. This will be set up for you to load in a 3d object and then map a movie to it or texture image. (if you don't have any 3d objects you can use either the couple that come with max or you might wanna check out turbosquid.com but make sure you get an obj)
hello i have been working on some thing like this, and my process is, model the scene in 3dsmax then use a light to make the shadows with the textures set to a grey, then save them out as a .mov. then use jitter to cross fade between them.
there is my first test, it works on a painting but the its the same idea