As some of you might be aware, under Settings->Storage, the iOS Spotify App has decoupled Downloaded Songs from the Cache. If one tries to look at the Library, there's no longer a selection for "Songs," the usual way to use the "hamburger" stack to remove a downloaded song.
Here is a solution:
Select "Your Library."You can then select between Playlists, Artists, and Albums. For the sake of brevity, stay in Playlists. To the right of the "Find in Playlists" search bar, there is a "Filters" selection. Click on Filters and select Downloads.
The filtered list will now display all of your Downloaded songs. Select each song or album and change the Download toggle from on to off. The app will ask you if you want to remove your downloads. Verify the removal.