This seems to be true across desktop environments.
Go to your songs and select a song to be removed. Click "Delete" on your keyboard and a popup should come up asking for confirmation that you want to delete it.
If you hit "enter" on your keyboard, the song gets removed and it starts playing. It should only get removed and should not interrupt the current experience.
Hey there you,
Yeah, you! 😁
Welcome - we're glad you joined the Spotify Community!
While you here, let's have a fun game…