I do not use the Spotify Connect feature at all however I fequently find it enabling itself and when I go to listen to music it will start playing on another device. I've dug through my account but can't find any way to turn this "Feature" off.
Hey @winterbikeridin! Welcome to the Spotify Community. ^^
This is not possible right now.
But there is already an idea for this going. Be sure to check it out and add your +Vote to get further updates on it.
And maybe it will get implemented soon. ^^