I tested out the laptop sound server idea. There's a lot of fiddling with audio you have to do to make it tolerable. For one, on the sound server you have to mute everyone else's stream or you get feedback. You also have to make sure that all of your players are using mic only recording, and all players should use headphones. Again, this is to prevent feedback, which can get quite bad (when the music stopped there was a constant hum that lasted until the other player logged off, due to persistent echoes).
Sound balance is kind of tricky here. I found that I had to have the sound turned way up on the sound server to get a reasonable result. If you take those steps, you get something that is workable. It does tend to have a lot of fizzing out, occasional sound glitches, etc though.