urmatter -- this should be possible.
I tried to do something similar awhile back, but found it tricky because the yaw from the magnetometer/gyroscope tends to drift a little.
What you need to do is after sending a [moveTo x y z yaw] command, send a [step x y z yaw] command where x and z are the coordinates from your vision system and y and yaw are from the drone's sensor data. The altitude can be gotten from the [sensor altimeter] command, and yaw via the second float in the list outputted from the [sensor gyroscope] command.
The implementation assumes that a yaw of 0 degrees is looking down negative z axis, positive x axis to its right, and a yaw of 90 degrees would be looking down the negative x axis, so you will need to initially offset whatever value you get from the gyroscope to make this match your setup. Additionally you'll need to compensate for yaw drift!
Be careful, the calculation is done via a PD controller, so if your yaw value isn't correct, it will move the drone the wrong direction and get worse and worse as it gets further away (most likely crashing hard into a wall!)
Cheers