So I finally figured out what is happening (after clearing, uninstalling, etc).
When I run Spotify on android, I get a notification that said "Downloading 0 out of 6 (0% complete). It never completes and it's impossible to get rid of.
It turns out that, even though I had checked "Hide unplayable tracks" and they weren't showing, this notification doesn't obey that. So they can't download, don't show, and you get a notification forever that can't be dismissed.
Workaround is to delete unplayable tracks from playlists, but shouldn't have to deal with that.