Removing background from live video

Hi there

I've already made a patch that finds the colours from a live video input. But what I'm trying to do now is remove the background from the imput so that you will effectively only see the person standing in front of the camera without the background, in order to somehow pick up the colours of any new objects introduced into the background, so that once a person wearing yellow for example steps in front of the camera, it'll pick up only on the yellow hes wearing and not the background.

I then want to send these RGB values to a Unity 3D model project that will change the colours of objects in Unity to that which the person standing in front of the camera is wearing.

So what would be the best way to go about removing the background? I've already taken a look at the jit.op object but couldnt really figure out how to use it.. Any help would be appreciated! :)