Cant control spotify from phone when using chromecast

[Ultinado]

My phone is a Sony xperia z5 updated to 6.01 and I just updated to the most recent version of Spotify. The problem I'm having is once I connect Spotify to chromecast I lose all controls. When I press anything such as shuffle on/off or next song it will momentarily disconnect my phone from chromecast (music continues playing) then reconnect with all the settings how they were. When I switch devices from chromecast back to phone the music will play on my phone but will continue to play on my TV as well.