true 360° video

Maybe use more than one camera that outputs a texture and then use jit.gl.pix with a shader to stitch the 4 (or more )inputs together ?
This would then be on the gpu not the cpu.

That is exactly what I did to create the linked youtube video. This does have the desired result for the reasons that I mentioned in my original post.

*does not

But thanks for the reply! Any other ideas?

Just to be clear, this is what I am going for. The cameras for the left eye (upper image) and right eye (lower image) move around a circle while rotating to capture each column of pixels. Both dodecahedrons are centered in the world, around the cameras. The purple one is just smaller and therefore closer so it shows up on respectively the right and left side of the green one due to the moving positions of the cameras. This cannot be done with stationary cameras.

I hope I have been able to communicate my idea clearly now, does anyone have an idea how to do this any quicker?

Afaik there are 3 options:

1. Render pixel by pixel as you are currently. Which is pretty much what you can do with raytracing. Or increase the patch size to multiple pixels... It might not be apparent in the result. Google scholar omnistereo for a method using cylindrical slices.
2. Render to cubemaps with each cube map meaning six renders with 90 fov cameras. You want the cube faces to have a higher resolution than your final image and/or use multisampling to eliminate aliasing in the final output. To get omnistereo you can render with an offset to the position for each vertex in the scene, but the stero disparity needs to peter out at the vertical poles. (And obviously this means one cubemap per eye). It works so long as your models have plenty of vertices. After the cubes are rendered there's a final resample pass to warp them to the 360 format e.g. equirectangular.
3. Do something clever with the depth buffer to offset pixels - this is how cheap plugins make regular 3d games stereoscopic. And what some stitching algorithms do. It requires some heuristics to fill in missing data.

Thanks! I really like your second idea, if I understand it correctly.
Do you mean instead of moving the cameras, I would rotate each vertex in the scene around the camera in opposite directions for each eye, the amount depending on its (horizontal) distance to the camera? That might actually work!

It does indeed work -- I used exactly this method in the infrastructure for the AlloSphere (a 3-storey spherical VR instrument with nearly total 360 stereoscopic coverage), and there are many projects in there using it. :-)

There's a part of the process here -- might be a useful source of the relevant GLSL code; the omni_render() function effectively replaces the projection matrix in the vertex shader:
https://github.com/AlloSphere-Research-Group/AlloSystem/blob/devel/alloutil/alloutil/al_OmniStereo.hpp

That is one awesome sphere!
I'm definitely going to try to implement this method into my previous rectilinear to equirectangular conversion patch. I'll see if I can use (but first understand - this is not my area of expertise) any of your code. Thanks! Btw I think there should be a tangent function in there somewhere to get the exact displacement for each vertex instead of an approximation.

I think it worked! I used your method of rotating each vertex in the scene instead of moving the camera and now I can render everything at once (or once per eye at least). I have also added the top and bottom cameras to complete the sphere and have less distortion during the conversion to equirectangular. Does anyone know a way to transform all vertices in a scene at once? Now I have to add a jit.gen to every object in the scene separately, would be nice to do that all at once.
Thanks!