hap is cool but unfortunately no option in my case since i work with frame sequences saved as .jxf-files.
i did however fiddle around with this a little in the meantime and at least in my case this actually does help: i write my clips in an uyvya-fashion to disk (beforehand) and then unpack/pack them into uyvy + alpha when reading. the performance boost is not huge but quite noticable - i guess disk speed is the bottleneck in my setup.
with 720p video a single argb-jxf is 3,7 mb vs. 2,3 mb employing this approach. the fact that the alpha channel is only half the resolution is - at least in my setup - not noticable.