I was told by support to post here.
It seems to be a common problem that Spotify cant cast to chromecast audio. I have seen numerous post on different forums with seemingly related issues. When casting fromspotify to chromecast audioi the music stops playing after a few seconds. The wifi network is the same. Works flawless on other apps eg google music. Same problem on different devices.
To me it seems like a server/app problem. Something for developers to look in too.