Kinect Projector Calibration

    Feb 25 2013 | 4:54 pm
    I'm starting a new thread on this, as I have a specific question that may need more attention:
    Has anyone implemented an "automatic" kinect-projector calibration like the one implemented in VVVV by these guys?
    I don't have a windows installation at the moment so I haven't tried it. But it seems to work fairly well (except that in the video he gets an calibration error above 1, which constitutes a calibration failure from what i've read elsewhere. The patch seems to wrap the openCV functions "cameracalibrate" and/or "projectorcalibrate" into vvvv.
    Breatheren has shared a library on the following thread which seems to wrap some of these openCV functions, but adapted specifically for an led-tracking application: (My first attempt at getting openCV installed on my mac has failed, but i'm willing to give it another shot if the tools can be adapted to kinect-projector calibration)
    There are other posts by Ad. that suggest there is work on creating what could be used as the shader part of the patch: ie, creating a warped cg shader for use in reprojection. there is also some helpful patches on in this thread on "undistorting" the kinect data here.
    Also this thread suggests lembert.dome was working on this:
    I also remember reading a diablodale post on the potential to implement this as part of his excellent jit.openni implemetation. (not sure where though)
    I'm wondering if anyone has gotten this to work in max. The math and code in this is over my head, so I doubt I could do it myself. But would love to do this in the max world. Are there problems/issues preventing people in the max world to come up with as elegant a solution as the vvvv work in this direction? Is there an advantage in vvvv (other than the patch being already available)?
    the patch ideally should project a checkerboard onto a piece of cardboard held up a various angles, sense where the cardboard is in 3d in each "snapshot", relative to the checkerboard corners, calibrate with calibratecamera, then create a shader that corrects the projection image accordingly.
    Is anyone close?

    • Feb 25 2013 | 10:29 pm
      Oh man, how much set up time this would save me... I have to calibrate 2 kinects and 3 or 4 beamers before every gig. I started looking into ways to (partially) automate the process but I got distracted by more pressing issues so I'm still doing it manually (physically aiming kinects and beamers and fine-tuning in Max). My tries didn't achieve much useful. Would love to participate in an initiative to realize this though I doubt my coding chops would cut it anytime soon.
    • Feb 26 2013 | 2:42 pm
      Dtr, what does your manual calibration consist of? In my manual calibration attempts, I've been positioning a gl camera to look at the kinect depthmap (warped mesh, among my attempts I tried your undistort patch) to a position (approximately) equal to the realworld position of the projector relative to the kinect. I then tweak the view angle of the gl camera to the projection angle of the projector. Though this works fairly well for most of the image, it doesn't work so well in the extremes, in the corners, etc... If I want to do a really tight projection map, I would need a better system.
      what is your manual process?
    • Feb 27 2013 | 4:50 pm
      Well... my setup and process is quite specific. I'm doing volumetric projections in fog, not traditional projection mapping on objects. See: I have 3 or 4 beamers set up in a circle, aimed at the center, and 2 kinects aimed at the center as well. I'm projecting openGL 3D geometries in the fog so that the intersections of the projections is the exact reconstruction of the virtual 3D geometry. Now to make it even more complicated, these projections are exactly aligned with my kinect-tracked body in the middle of the space. So in a way it is projection mapping.
      Still following...? I've wanted to make a technical documentation video of this for ages ;)
      My calibration process consists of: - position beamers and camera's in the space using a measuring tape - physically align the beamers' projected grid to measured points on the opposing wall and/or to a pole in the center of the space (which has to align with the center of the openGL space), i usually use my camera tripod for that - physically align the 2 kinects to the center of the space - then the software alignment starts which is tweaking kinect skeleton tracking and openGL projection parameters so that the tracked joints of my body re-projected in the space fall more or less exactly on my body
      I try to get an error margin of less than 2 or 3 centimeters. The way I'm doing it is obviously tedious and inaccurate. There's a tremendous potential for automating some steps...
    • Feb 27 2013 | 6:45 pm
      Hi Dieter, Thanks for sharing your calibration process... It isn't too far off from what I'm doing, though my mapping is on the dancer/walls/floor and from only one point of view (one beamer, one kinect).
      Also, thanks for sharing the video documentation of the project. This work seems really lovely, I'd love to see it live. Very innovative.
    • May 23 2013 | 12:34 am
      This topic is popping into heads. The knowledge and tech (e.g. ROS, openCV, RGBDToolkit, Chips, etc.) is out there. Using b/w checkerboards w/ 3d shapes to align color and depth cameras by deriving intrinsics/extrinsics and like methods. I have also seen a few demos which can do fast automated aligning of camera and projectors using quickly flashing gridpatterns and lines to derive projection surfaces,angles, and their relative sizes.
      What it seems we want, is someone to write the code and wrap it into an external. Or expose it with OSC so we can in/out it from Max. The OpenCV project provides the bulk of the work, code, even DLLs. If you want .NET,Emgu wraps OpenCV. That level of accuracy may not be what we all need. For example...
      Recently, I've been working on some private updates for dp.kinect for aligning Kinect(s) to the real world which then I related to the OpenGL world in a 1:1 manner with a technique not OpenCV. Most of that work is complete. I hope to get a few answers from the Kinect team to better approach the last 20%. Then I can see if this approach does a good-enough-job. So far, it doing what I need.
      This code uses the Microsoft Kinect SDK so it will be Windows only. Why? The OpenNI side is sketchy (1.x API abandoned, 2.x API in beta, no known 2.x API Kinect hardware drivers for OS X). And a caution. I haven't written jit.openni which hacks the Kinect. Instead, I've written the external for the OpenNI 1.x API which is hardware agnostic. Someone else has written OpenNI hardware drivers for various devices like the Asus, Kinect, PMD, etc. This OpenNI Kinect hacked driver is failing on newer Kinect models (there have been silent model updates).
      Separate from aligning projector(s) to the world. What I did last year for a  dance troupe, was model everything in real-world. I'm tending to do everything that way now. I got a a measuring tape and building plans. I created the 6 walls/ceiling/floor of the performance hall and the stage in OpenGL using 1:1 scale. Then, moved my to the approximate location of the projector. After that, I would (in OpenGL) make a grid pattern appear on the stage and backwall...which then appears on the real stage. I could then tweak the projector's zoom, lens angle/position to get my grid to align accurately on the stage. I remember once, a dancer wanted an image adjusted. I asked how much, she said "about here on my leg". I used a tape measure to see where on her leg it was, then typed in the real world measurements into my jit.anim.path and it was perfectly where she wanted it. It took more time for me to get out of my perch->stage->measure->perch than to make the actual code change.
      I am unclear if projector alignment of OpenCV will be doable when you are projecting on to a 15m surface. How big a checkerboard does that mean I have to construct? And how do I move around in the air and keep steady that massive checkerboard? My manual alignment was perfect in the middle of the stage; on the outer edges it was off maybe 10cm. I believe that I can improve on that by using cornerpins (I didn't use cornerpins w/ the dance troupe).
      I do see value in OpenCV's code for OpenGLProjector alignment, but do we all really need that?
    • May 23 2013 | 9:07 am
      Not all but some do ;)
      I'm following a similar manual procedure, described above. Thing is, with 2 kinects aggregating into 1 model and 3 or 4 projectors in a circle volumetrically reconstructing that 1 model it becomes a tedious job...
      It's not a necessity but some automation would lighten the job for sure. Because of the elaborate physical + digital setup work I am uncomfortable building up and performing on the same day. I always request to build up a day beforehand. Sometimes that's not possible and it makes for stressful sessions. Not the best circumstances for having a good show. The disadvantages of being your own tech crew...
      What's holding me back from cooking up a solution is limited coding skills. I tried delving into PCL's (point cloud library) people tracking. I thought if I can use PCL's tracking instead of OpenNI I can then also use their calibration functions. Didn't find my way through all the dependencies, compiler settings, etc. Didn't help that it was in beta and on Linux either. Perhaps the more established openCV would be easier to tame.
    • May 23 2013 | 8:42 pm
      FYI, I learned today that the Kinect's color camera, depth camera, and accelerometer all vary their readings based on temperature. The Kinect SDK writes about this in the accelerometer section. Additionally, there have been at least two papers, here is one, which document the shift.
      With test subject at 1.5m, from a cold start to 60minute warmup, the RGB could vary by 1cm and depth by 2cm. Those numbers get larger as your test surface is further away from the Kinect. So for any OpenCV type calibration, keep the Kinect active and working for 60 minutes to warm it up and then do the calibration. And then later, always do the warmup before you actively use it.
      The accelerometer has a 1 degree rotation accuracy. Then, a +/-3 degree drift based on temperature warmup. If you were to based XYZ values on the accelerometer, they could vary as much as 28cm from startup with a test subject at 4m. Luckily, the drift is predictable and unique to each Kinect. So like with OpenCV, it can be measures and then values used to adjust. That leaves only the 1 degree accuracy value which with a test subject at 4m can vary 7cm in Y.
      The paper recommends a 60min warmup period for calibration/usage. Always keep the temps the same. Finally, to keep the Kinect away from air drafts as such drafts can rapidly (in seconds) cause the readings to change.
    • May 25 2013 | 12:22 pm
      aargh... even more parameters to factor in!