combating random access to long video files...

I am building a video sampler which works, but has a lot of potential to be enhanced.
First of all the biggest problem is fast access to a video, and I am using the techniques of the VPS patch of Andrew Benson.
I use the loadram function to jit.qt.movie to load a portion of a video into ram, and I use the import message to load the same portion of sound into a buffer~.
I choose my own audio handling, as the preparation for a spigot~ based playback takes much much longer than importing sound into a buffer~.

The problem I face is that importing the sound into the buffer~ takes very long. It seems related to the start point of the portion I want to load and/or to the overall length. I didn't want to cut the original video into thousand pieces and rather access it randomly.

How could I get these bits faster loaded? I would expect that it should be at least as fast as the length of a cue, which are set to a few seconds. but it seems the further away from the beginning the longer it takes to load them - up to minutes...

Any help is appreciated...

Stefan