Mixer with master Faders

Hey this may seem pretty stupid but I currently have 12 separate faders that control each individual channel of audio these are all linked to a master Left and a master right fader, however when all of the 12 faders are down there should be no audio being output if the master faders are up only when you slide one of the 12 faders up and the masters are up then you should have sound. the issue i'm having is that when all of the 12 faders are down when the masters are up audio is still output, i understand that it's because i'm sending a value of 0 to the master faders and then this is then being gained by the value set, how can i get it to only play the track if the fader is up along with the master like a normal mixing desk.

my signal flow is sample to the small fader then to the master and then the output, what can I do to get around this issue?

Thanks