I might start by reading some of Robert Rowe’s books. Specifically, Interactive Music Systems and Machine Musicianship. See: http://homepages.nyu.edu/~rr6/ Also, look into George E. Lewis’ piece called Voyager. The approach many have taken has been to break "style" down in to a number of parameters that are easier to quantify. "modality", "tempo", "density", &c. I would start simple and build up complexity until the result started to become satisfying.