I was trying to find the link, but I couldn't, but I've seen an Arduino-based system that exists as a matrix of patch points that let you dynamically reroute computer programs. One of the examples was to glitch out super mario bros.
From what I remember it just made binary connections which controlled a software matrix via OSC. So something like that would be possible, and is something I was thinking about the other day too. I'm sure there's a way to create a matrix like that where you have unique resistance values setup in a way where you can tell what connections are being made to what in an arbitrary way.
Inside Max you would then just use [matrix~] to make and create those connections without audio dropouts (which might happen if you script audio cable connections.