On PlayStation 4 when I use shuffle on playlists which have songs from multiple artists it only plays one artist at a time.
For example I have a Rock-playlist containing Disco Ensemble, Billy Talent and Offspring.
When I hit play, it starts playing Disco Ensemble's first album in order of the tracks.
When I hit shuffle it shuffles the tracks but won't mix the bands.
For some reason it plays all the Disco Ensemble's tracks first (in shuffled order) until it runs out of their tracks, and only then moves on to Billy Talent.
Then it plays all the Billy Talent tracks (in shuffled order) and only after that moves on to play Offspring.
So it is not possible to listen to one track from one band and the next from another one without manually choosing them.