It looks like it might not necessarily be Spotify's fault. After noticing that Google Music and iTunes are now doing the same thing, I did a little digging and discovered this footnote within the iOS developer docs for the idle timer setting:
You should set this property only if necessary and should be sure to reset it to false when the need no longer exists. Most apps should let the system turn off the screen when the idle timer elapses. This includes audio apps. With appropriate use of Audio Session Services, playback and recording proceed uninterrupted when the screen turns off. The only apps that should disable the idle timer are mapping apps, games, or programs where the app needs to continue displaying content when user interaction is minimal.
So it seems that this is a directive from Apple that Spotify is complying with, along with the developers of other popular audio apps.
Probably the largest annoyance about this feature is if you have a Wifi speaker like a Sonos or Bluesound that you stream Spotify to (as I do). Whenever I pause the music, it will disconnect from the speaker. I then have to go back into the app, tell Spotify to reconnect to the Wifi Speaker, and then hit play again. Not an ideal user experience at all, as I'm on and off the phone all day long in my home office, so have to pause the music each time I receive a call. Incredibly annoying, whereas previously it's plugged in on my desk; pause and play were a single easily-accessible button push.
Re: iPhone screen locking while playing and connected to charger
I tested it with the new Spotify update. Unfortunately, the screen still dims and shuts off when connected to a charger. The only reasonable solution I found was to set my display to never turn off. It works ok, but occasionally I forget to manually turn the display off and I find my phone was on in my pocket.