Here's a few tips that I've uncovered to solve the issue.
Firstly, find the app named NetGuard, install it and set it to block all cell data usage from Spotify. Just click on the angled Cell Signal Strength icon next to Spotify, so it turns red with a slash, and you're all set. I've found this app so valuable is finding scores of other 3rd party apps sneaking cell data that I paid for the full version to create logs of all apps.
Next, send an email to Spotify support at firstname.lastname@example.org. Tell them of the situation, and send them screenshots of the Cell usage, and the NetGuard logs. After I did, I found within a few days they had it resolved. I'm still tracking it, and blocking cell, but I haven't had an attempt for almost 2 weeks now, where before, it was only using cell, even when connected to Wifi.
Well, if it's not working while being restricted to Wifi, and your phone is connected to Wifi, then you need to double-check your settings. If you can't figure it out, you need to contact support. Ultimately, it appears it was set to run on Cell only. Check the settings.