Implement an actual shuffle function

[RoninTheOrigina]

Yes, you read that right.  The "shuffle" function in Spotify is nothing more than a randomizer.  There's a difference?  YES.

RANDOM - Play one song.  The next song can be ANY song in your playlist.  Including the one you just listened to.  Though I've never had this specifically happen, I literally just heard the same song that I heard 2 songs ago.  This is how Spotify's "shuffle" currently functions.

SHUFFLE - Take your entire list, shuffle them (like cards), and play the first song in the shuffle.  Once finished, move that song off the pile.  Once the pile is depleted, shuffle the deck again and start over (maybe pop a message up).

I know enough about programming (not much) to know that shuffling is probably a bit harder than random, but come on.  I may be in the minority, but I don't have a lot of little playlists.  I have one list with nearly a thousand songs in it, because I like all the music, and am rarely not in the mood for one of those songs.  But what I can't stand is hearing the same song over and over again.  I think there may even be songs in my list I haven't heard yet, while I've heard others 4-5 times.

I know it may not matter much, but I would probably lay down the money for at least a year subscription if this feature was implemented.

EDIT: Another user pointed out that I can see what all is queued up in my shuffle by clicking the Play Queue link, and that Spotify supposedly sets up a rolling shuffle of 50 songs.  It would be nice if this range could be expanded to say 50-100% of your playlist.

EDIT 4/6/12: I just dealt with the same song 16 tracks later.  The song played the first time yesterday afternoon.  I went home, listened for a bit on my home PC (just a few songs), then started listening again this morning.  If the list recycles itself after a day, or if you log in from another computer, I see that as an area for improvement.

[brandynhazelton]

Spoiler

Im not sure, i tried before and they closed it and referred me to comment here.

[5m3cvmdlkr174mw]

I don't know if that's part of the recent changes, but I love how shuffle creates a random order, but sticks with it. What I mean is that with some apps the order of a shuffled list changes every time you skip a song. Instead you guys create a random order on shuffle and keep it until I shuffle again. I love that because I get to hear my entire list even if I skip.

[Samahlam]

Apple ITunes/Music app shuffle is the hallmark standard Spotify should try to emulate...when I shuffle with Apple, it literally plays a random song and the whole list order changes completely...I press shuffle on Spotify and it just plays the songs in a pre determined order no matter how many times I take shuffle off then put it on. It can't be that hard to make the shuffle function completely random...Such a bug that I'm thinking to cancel my subscription

[bytebodger]

(TLDR: This can be fixed with the API.)

I only became a Spotify user (and paying Premium user) several months ago.  Overall, I love the service.  But their inability to implement a simple SHUFFLE (and "shuffle" is NOT the same as "random") feature was driving me nuts.  It particularly bothered me because their own verbiage says that you can "shuffle" - but their shuffle feature really does no such thing.  I have a working solution to this problem on my local system.  It probably won't help many people right now (unless you're a programmer like me), but I hope to get my utility on a public website (and/or an app) soon so others can enjoy the fix without having to learn code.

There are several problems that I've found with their faux-shuffling methodology:

1. The "shuffle" feature is just their way of building a new, temporary playlist from a larger one.  That would be OK, except that their shuffled playlists are capped out at 80 songs.  So if you have a playlist with 1,000 songs, it will "shuffle" you a new sub-list from that master list, but the new sub-list will only contain 80 tracks.  And when it reaches the end of those 80 tracks?  It seems to go back into a reshuffle mode.  So there is a very real chance that some (or many) of the tracks from that initial list of 80 are represented again in the newly-shuffled list.  This leads to you hearing the same songs frequently.
2. What's even worse is that it doesn't seem to truly "shuffle" from the master list. Instead, it takes the current song that you have queued up, and uses that song as a seed to determine what other songs should be included in the abbreviated, shuffled playlist.  So for example, if you sometimes take your player off of shuffle mode, and then flip over to a new playlist, begin playing that list, and THEN turn on the shuffle feature, it will always use that first song in your playlist as the "seed" to create the new sublist of faux-shuffled content.  So unless your playlists contain only songs that are nearly identical, there's a good chance that "shuffled" lists it creates for you on the fly are overly-weighted to sound like the first song in the list.  I saw someone else in this thread complain that her poly-genre playlist that was supposed to please everyone in the office would only play rock music.  This makes sense in light of this point.  If the song she was playing (or the first song in the playlist) was a rock song when she put it on "shuffle", their BS algorithm then looked at that one rock song and tried to find other rock songs in the playlist from which it could build a new Play Queue.
3. I can almost prove that their faux-shuffle algorithm also tries to weight the selections according to how much you "like" them (in other words, how often you play them, or how often others play them).  The MAJOR problem with this approach is that it becomes self-enforcing.  The early songs that you added to your playlist will seem to be popular (to their algorithm) because you've played them more.  The new songs that you add to the list don't look as popular, because you haven't had any chance to hear them in the (rigged) rotation yet.  What is my evidence for this claim?  I have a playlist with fewer than 200 songs in it.  I just added a new song to it today.  Then I put the playlist on "shuffle" and looked at the Play Queue and the song was nowhere to be found.  This, by itself, is not surprising because the Play Queue only captures 80 tracks from my overall list of 200.  But then I started toggling the "shuffle" feature on and off and on and off again.  This forces the algorithm to run again and it rebuilds the list of tracks in the Play Queue.  No matter how many times I did this (and I probably did it several dozen times, just to prove my point), the new track I added NEVER showed up in the Play Queue.  The songs in the Play Queue DID change when I toggled "shuffle" feature on/off repeatedly, but the new song I added simply would not show up in the queue.  Either Spotify hates this one new song that I added to the playlist and refuses to add it to the "shuffled" list, or it's being excluded for more empirical reasons.  
4. If they are weighting shuffled Play Queue additions by "popularity", it's a godawful design choice.  If you love Nirvana and you've put 40 of their tracks in your playlist, you'll get bombarded over and over again with Smells Like Teen Spirit, but you might have to wait months before you ever hear Polly.  

In the next comment I'll explain how I fixed this...

[bytebodger]

(TLDR: This can be fixed with the API.)

Spotify offers a fairly-robust API (Application Programming Interface).  It's easy to use (if you're the coding-type) and it's pretty well documented.  The API offers you the ability to actually push songs/playlists to your live music player in real time.  It also allows you to query all of the existing playlists and their accompanying tracks.  So here's what I did:

1. On my local machine, I installed the free PHP SDK (Software Development Kit) that's available on GitHub.  I then registered for a "developer account" on Spotify (it's free and only takes a minute or two).
2. Once I had the basic handshake (login) working via the API (this probably took me 30 minutes to get working), I pulled back all of my existing playlists (I currently have about a dozen set up).  
3. Once you have the playlist, you can then pull a list of ALL the tracks that live under that playlist.  I simply dumped them into an array and shuffled them.  (Note:  I did a simple and true shuffle.  Not a song-by-song randomization.  If we use a deck-of-cards analogy, this means that I literally shuffled the songs in the playlist as I would shuffle cards in a deck before playing a hand of poker.  The overall order of the tracks is changed every time - but there is only ever one representation of each track during each go-through.  You wouldn't want to play a hand of poker where the ace of spades could, theoretically, be dealt out two or three - or more - times in the same hand.  And I don't want to list to a "shuffled" playlist where the same song could be played two or three - or more - times in the same session.)
4. Once you have the newly-shuffled (REALLY shuffled) list of all the tracks in your playlist, you can push them back to your live Spotify player as a new Play Queue.  And they will ALL play in the completely-randomized order that was created from your code.  This means that once you've heard Smells Like Teen Spirit in your Totally Awesome Grunge playlist, you will never hear it again until either 1) you switch to a new playlist, or 2) you listen to all the songs in the playlist and you restart it from scratch.
5. What happens if, for some reason, I don't like the truly-shuffled order of tracks that was created by my homemade shuffler?  Simple.  I just rerun the web page that launches these API calls and voila!  I have a newly-created, truly-shuffled version of the same list in the Play Queue of my Spotify player.
6. There seems to be a hard-limit on the number of tracks that Spotify will show you in the player.  That limit is 80.  But I have confirmed that if I push a playlist to the Spotify player with, say, 300 tracks in it, it will keep all 300 of those tracks in the (newly-randomized) order that was generated in my code.  So when the Play Queue reaches the 80th track in the list, it will reload the Queue with the next 80 tracks from my manually-created, truly-shuffled list.  But since it's not doing a "fresh" shuffle from scratch, this means that none of the tracks in the first group of 80 will ever show up in the second group of 80 (or the third, or the fourth, or the...)

I know that this comment isn't immediately-helpful for the "average" user just trying to play their Spotify music without becoming a software engineer in the process.  But I wanted to throw this up here to benefit anyone who knows how to leverage this knowledge before I can create a publicly-accessible solution, because I know that this is a huge pain-point for many users and Spotify doesn't seem to care about it in the least.  

On one hand, it's nice that they have a robust API that will allow the technorati to "fix" this issue.  On the other hand, I really consider this to be a massive blunder on their part as a company.  This is such a simple feature and people have been clamoring for it for YEARS.  The answers to your product's shortcomings should never be, "Here.  We created an API for the tech-savvy to use.  Now go fix it yourself."

[bytebodger]

A few final notes about the solution I've outlined above, then I'll stop spamming the group.  😄

1. This solution has no impact on your playlists.  It doesn't change them in any way.  That's because the API call is building a new Play Queue from the playlist.  (The Play Queue seems to function as a temporary, ad-hoc playlist.)  There are API calls that allow you to actually change the playlists themselves, but you wouldn't (shouldn't) need to use any of them to make this work.  
2. It took me about two hours to get a basic version of this working on my local system.
3. There is no "memory" of these ephemeral Play Queues - neither in Spotify nor in the proof-of-concept code I've created (although it could certainly be implemented with more work).  This means that if you use my API tool to launch a newly-shuffled playlist in the Play Queue, all will work as I've described it above.  But if you then flip back-and-forth inside the Spotify player between different playlists (which, in turn, will repopulate the Play Queue), and then you use the API tool to create another Play Queue from the original playlist, it will build - from scratch - a new list from the master list of songs, whether you heard some of those songs a few hours/days ago or not.  As long as you leave the generated Play Queue active in the Spotify player, it will play a truly-shuffled docket of all songs in the playlist - playing each of the songs, in random order, once and only once during any given session.  
4. My "workflow" now for playing music in the Spotify player now looks like this:  I open the player.  I pull up my web utility on my local machine.  It populates a dropdown list of all my existing Spotify playlists.  It choose one from the list and submit the form.  The code/API then automatically loads up all the songs from the playlist, shuffles them, and automatically starts playing the shuffled list in the Play Queue.  That's it.  I don't need to do any additional coding or technical tweaking to launch a truly-shuffled version of my playlists whenever I want.

[chrissound3]

bytebodger do you have the code on Github or something? Do you need an API account?

I'd be very interested! Thanks for posting.

[rednblu]

Yes!  Go team!

Great explanations, @bytebodger!  Thanks for all three very eloquent, very concise, and very clear postings.

And Go, @chrissound3!  Do it!  Come back and tell us what you find!  What you report will be helpful to all of us MusicListeners who try not to do the programming that Spotify should do for us.  But we will make the "programming fixes" ourselves if we have to do that-- to hear the gorgeous Spotify music!

My friends gave me similar "programming fixes" for Shuffle of over 100K tracks (over 100,000 tracks) that works completely in an Excel spreadsheet and other Windows "documents"-- So that if Spotify makes the API solution unstable, we can migrate to some other streaming service with the working "solution".  We will hang together!

Spotify is the best, right now!  Thanks, Spotify-- for keeping the streaming service as stable as it is.

God help us!

[bytebodger]

OK, here’s one way to prove that Spotify's "shuffle" feature is no such thing:

1. I created a playlist of 99 popular rap songs. You should be able to access it here: spotify:user:12139360648:playlist:19bWMy5OSgPuCdQfGAsMwW
2. Then I went to Forgotify and grabbed the first track it came up with: spotify:track:50CQRXMGKilUaa00o8pyua (Carnaval de Guaranda by Eduardo Cadena)
3. Since that track came from Forgotify, we can safely assume that it has massively LESS popularity than all the tracks in my original playlist of 99 popular rap songs.
4. This means that there are now 100 total songs in the playlist.
5. I opened the playlist and began playing it with shuffle toggled on.
6. Then I went to the Play Queue screen and looked at the first 80 songs generated in the shuffled playlist. Carnaval de Guaranda was not in the list. If I played the last song in the Play Queue (thus forcing it to reload the remaining 20 songs in the list), Carnaval de Guaranda DID exist on the second page, near the end of the list.
7. I toggled shuffle off-and-on-again (thus forcing it to build a newly-shuffled Play Queue from the playlist).
8. Carnaval de Guaranda is not in the first page of results. It IS on the second page, near the end of the list.
9. I repeated this test 50 times – each time forcing Spotify to “reshuffle” the 100-song playlist.

My newly-chosen, purposely “unpopular” track (Carnaval de Guaranda) was never listed higher than 77th in the shuffled list. Out of 50 trials, Carnaval de Guaranda was presented, on average, at the 87th position in the shuffled list. That’s not random. And that’s not what most people think of when they talk about “shuffling”.

So what difference does this make, as long as the song does come up SOMEWHERE in the list?? Well, if you don’t listen to the entire playlist, all the way through, you’ll likely never hear Carnaval de Guaranda. And if this “unpopular” track is always shown near the end of the shuffled list, that means that significantly-more popular tracks will always be shown nearer the beginning of the shuffled list. So you'll be more likely to hear them during an "average" listening session.

People rightfully complain that Spotify’s “shuffle” algorithm stinks. If you have sizable playlists, you’re usually not going to listen to the entire playlist in one session. You’ll listen for a few hours to one playlist, then for a few hours to another, and then maybe back to the original. But when you go BACK to the original, you’re starting back at the beginning of a newly faux-shuffled list. So you end up hearing a much stronger concentration of the same songs – even when you have a very large playlist.

Even if you did listen to the entire playlist all the way through, do you always want to hear the “unpopular” tracks near the end of that session and the “popular” tracks near the beginning?? If you added it to your own playlist, it’s probably “popular” in your mind.

[CrapuleJack]

Just try to shuffle on artist "Rancid"; no matter how much skipping you do there are whole albums that will never appear in the list even one song.