Is there a way to rotate an object not by the center (as rotate or rotatexyz messages can do) but like you open a door for example ?

rotate-opening-a-door

another example, jit.gl.text3d allows that with "align" message.

Here's an quick example achieved by rotating the render context. Doing this with object-space transformations is very possible, it just takes a bit more time. I don't think there's a built-in way to offset the rotational center of an object, so you'd need to use a bit of trig and simultaneously rotate and reposition. I can do this later maybe when I have more time. Or perhaps I'm wrong about there being no built-in way??

Here is the more subtle solution - rotating the object relative to its own object-space.

It is exactly what I need to do, especially your second patch.

I had a look at jasch's [3dmatrix] object too, but I can't figure if it could be used with open GL.

Yes, 
It is exactly what I need to do, especially your second patch.

it is for a theater project where 3 screens could move back and forward, like a door.

In fact, 
it is for a theater project where 3 screens could move back and forward, like a door.

rotate opening a door