Agreed, it's why I personally use Ventrilo to host my games instead of Roll20's built-in voice. The PTT feature in Ventrilo means you're only transmitting when you mean to and everyone doesn't have to listen to everyone else's distracting background noise.
