I know this idea is similar to an idea presented earlier, that were closed.
Instead of just greying out tracks and albums that aren't available for streaming (which I agree would be bad for the user experience), it should be possible to have a short bullet point list on the artist page presenting the entire discography.
This way it wouldn't be necessary to double check with wikipedia etc. to find out which albums are not on spotify.