I think the reason it shows up as an option when connecting using Guest Mode (via PIN) but not over WiFi is that it is only possible for the app to determine if your Chromecast firmware supports Spotify when connecting over WiFi. When connecting via Guest Mode it appears to only be able to determine thise once it has actually connected. You gest a notification appears saying that it is connected but it gets disconnected almost immediately once the app determines that Spotify isn't supported.
So we just need to either wait for the firmware update to the original Chromecast (which is aparently coming in the next few weeks) or buy one of the new Chromecasts (I've actually gone for the latter as I was already considering getting a second Chromecast for another television in the house before the new Chromecast was announced).
I have this same problem. I'm on a 3.8 version of spotify and my old chromecast firmware is 1.16.444433. Anyone else still have this issue? I can see the chromecast in guest mode, but it doesn't actually connect properly.
I'm hoping I'm just due for another firmware update, but my impression was this was the most recent one...