For windows, I recommend using jit.dx.grab instead of the QT version you have. In general, there is richer support for DX than QT on windows.
If you use jit.dx.grab and qmetro, do you have the same problem? If you do, look at the Extras meny in Max for JitterTester and try to get the camera to work there. You might discover a setting/option that you need to use in your patch.