[All Platforms] Consolidate duplicates

[Kerumen]

It would be nice if Spotify can recognize identical songs to avoid duplicates in your Liked songs and playlists.

This is pretty common for songs released first as a single then within an album.

For example, if you like the single one, the UI will not show the song as liked when you browse the album. And if you like the song in the album, you'll end up with a duplicated version in your library.

This also happens a lot for songs that you listen in Daily Mix or Discover weekly. If it's not the exact same song you have already liked before, the UI is inconsistent and you can save a duplicate.

Apparently, there is already an internal merge mechanism as the play counter is not split between the different versions of the songs. Maybe this can also be used to consolidate the UI.

Repost of https://community.spotify.com/t5/Closed-Ideas/Your-Music-Remove-Duplicated-Songs/idi-p/4546787 which didn't get enough kudos.

[user-removed]

This exact same thing happens to me, it's extremely confusing. Glad I'm not the only one with this issue. Has anyone figured out a fix for this? Sometimes I notice it on deluxe albums that are no longer offered but I saved when they came out.

[rednblu]

Simple would be quicker.  Just DeDuplicate by TrackName-- by clicking "DeDuplicate" that developers would add to the RightClick menu of both 1) Folder and 2) Playlist.

Here is an example playlist to be DeDuplicated; here are the songs in that playlist.

Billie Holiday – I'll be Seeing You
Billie Holiday – I'll Be Seeing You - Live (1956/Carnegie Hall)
Billie Holiday – I'll Be Seeing You - Bonus Track
Billie Holiday – I'LL BE SEEING YOU - Original
Billie Holiday – I'll Be Seeing You - Remastered

DeDuplicate steps:

• Cut Artist from the beginning.

• Cut everything after the TrackName, including "Live", "Bonus Track", "Original", "Remastered".

• Reduce the TrackName to all lower case so that "I'll be Seeing You" will be recognized as a duplicate of "I'LL BE SEEING YOU".

• Cut off everything in parentheses ( ) or brackets [ ] after the first position-- thus removing the editorial comments that have nothing to do with the name of the song.

• Delete all punctuation such as ', ?, /  -- because if you scan playlists ordered by Track, you will see that "duplicates" often have different punctuation.

• Delete all spaces -- because often "duplicates" have extra spaces-- or missing spaces-- between words.

• What is left is a NormalizedTrackName, URL_Index.

• Order by NormalizedTrackName.

• Delete all but one record from each group with duplicate NormalizedTrackName.  I have done all of these above steps in an Excel spreadsheet if anyone is interested.  DeDuplication makes my playlists ten times more impressive-- particularly at parties.

Where you will want some "duplicates"-- such as brilliant covers and even more exciting "live" performances-- the simple solution would be to add the desired duplicates to a separate playlist that you would then combine with the DeDuplicated big playlist in a Folder that you would "Play."  

Also into this "duplicates" playlist you would add those rare tracks where completely different songs end up having the same NormalizedTrackName by the above process.

No Spotify database clean-up of bad data is required; no database restructuring is involved.

The above DeDuplication process removes 99.9% of the hated duplicates that drive you nuts when you are in the middle of the pleasantries.

[nintynuts]

@rednblu

Your suggested system is treating the symptoms, not the problem. Also, spotify want to keep their client thin and have all the logic being done server side, so architecture-wise it wouldn't fit. The idea of my system is to allow people to find ONE VERSION of the song they want, but support all the variations in a structured manner. It would then accurately be able to reflect the popularity of songs, as well as whether an artist is any good live, and provide better association of cover songs (decluttering 'as made famous by' and 'in the style of' from their track names).

The digital fingerprinting in my suggestion is done as a last resort to reinforce the decisions made by analysing the metadata of the song to reduce the number of comparisons that need to be made. 

The benefit of this optimisation is not only good for users, but it would probably halve the number of songs spotify need to keep on their servers. 

A good example is Sunday Girl by Blondie. there are about 40 tracks listed when searching 'spotify:search:title:"Sunday Girl" artist:Blondie'. Just looking through these manually, there appears to be about 10 versions, which are:

- The original

- 3 remasters

- the french version

- the live version 

- 2014 re-recording

- 2 BBC radio 1 recordings

- 1 mix

About half of these seem to be the original, but without fingerprinting you can't be sure. 3 are 10 seconds shorter, and may be the french one, while one other is 15 seconds longer and is live - but I could only tell by listening. This is a potential 75% saving in required storage space, assuming spotify doesn't have any file system based compression. Less popular artists will have fewer duplicates, as their tracks won't keep getting re-released...another metric to measure popularity by.

[delae]

Peter, that's exactly the problem: multiple track album version. Logging off and on will not solve this problem. And it's annoying.

F.e. I marked some tracks from this album as "Saved" some time ago: spotify:album:3uocz31s0fmZ8SRP1rCM8e

But it seems this album got replaced in the mean time by this version: spotify:album:7o8Pkxy7OmloQ1atWwlfBX

So when I navigate to this album using search, I won't see my "Saved" marks. This is not a exception. It occured already multiple times to me.

Please consolidate album versions if it's really the same album, or show different realeases if they really are different.

There's a similar request: https://community.spotify.com/t5/Spotify-Ideas/Consolidating-duplicate-tracks-which-appear-on-multip...

[Weidick]

This spring there apparently was implemented a function to prevent duplicates in playlists.

And it works!!!

But when is a duplicate not?

It only works if you try to add a song, that is completely the same as another in the playlist. That means same song, same artist, same album. And maybe same something more.

I frequently take all songs from all albums from a certain artist or group and put them into a playlist. And the "duplicate function" does not work here. Only if i take the same album twice. And i would not do that.

I have not any answer for how to do the trick automaticly. 

The only cure i found was to sort the songs alphabetical and then browse thu them an remove the unwanted.

But i have a dream. That it will be possible to do it automaticly.

[rednblu]

How about

* Song

and

* Song - Live

and

* Song - Remaster

Are these Duplicate?  Should the automated DeDuplication keep only one of the above?

[Weidick]

Dear rednblu

Excatcly

I dont know how to do this, so i agree that it is almost impossible to program.

In the old days classic music had BWV and köchel numbers and in that way they were unique regardless whom played it.

Maybe that is the only way for modern music as well.

But we dont have these numbers, so i cant see the solution.

If it is possible to do the thing you suggest, it might be a step on the way.

But thanks for considering

Best regards

Poul Erik

[Marco]

Weidick and rednblu - I've moved your discussion over here to have that in one place. 😉

[rednblu]

Dear Weidick,

Of course, Spotify could give us an automated solution-- but they won't. 🙂

Thanks for helping me improve my *design and *implementation of my programming Workarounds to remove Duplicates so that I don't get ticked off at Spotify. Most of the obvious duplicates I now remove with a programming workaround-- then I have an acceptably easy Workaround way to use the 0.9.x version of Desktop Windows Spotify to remove the other duplicates that I don't want.

What I end up with is a >> *Randomized playlist-- << that you can copy, @Weidick, to your own PlaylistName and run the following routine yourself. I repeatedly Cut and Paste batches of 300-400 tracks from that *Randomized playlist to the "yellow" editable PlaylistQueue in 0.9.x Desktop Windows Spotify by the following.

* Select a "nice place to start" in that playlist.

* Do Shift-PgDn-- maybe 6 to 15 times-- depending on my mood-- to highlight a bunch of tracks.

* Then I do the "Cut and Paste to the Queue" operation by the following: 1) Right click on the highlighted bunch of tracks; click on Queue from the dropdown to do the Paste; then hit the Delete key to complete the Cut of the highlighted tracks.

The programming to generate the *Randomized playlist I have designed, with your thoughtful assistance, @Weidick, to work inside Excel 2007-- just for a convenient Workaround. The only "duplicates" in that *Randomized playlist are the "duplicates" that I want-- for example, I have everything by Thea Gilmore and Phillipe Herreweghe even if the performance is a "cover"-- but with the Excel "program" I have removed 99% of the unwanted duplicates that originate from Spotify copies *on different CDs or *Spotify data errors from misspellings in TrackTitle or crazy variations of Unicode apostrophes, Escape code clauses, or missing spaces between words.

If you try "Cut and Paste to the Queue" from your own 0.9.x copy of that *Randomized playlist, you will get a wide range of every kind of music that there is. I operate from an >> automated DeDuplicated MasterFolder << with 60,000 and growing numbers of tracks, which MasterFolder contains Zero unwanted Duplicates-- by my own wishes-- and made possible with the Workaround programming-- suggested by you, @Weidick-- to make the *DeDuplication and *Randomization mainly automated so that Spotify with all of its peculiarities, irrationalities, and "What! What where you thinking!" destructive Spotify release cycles makes me smile with 99% Customer satisfaction-- because Spotify now gives me the music what I want *Truly Randomized and *DeDuplicated by my personal criteria.

@Marco-- Thanks for moving the fragments of this conversation to a centralized and useful "library" location!

[jamiegray]

good idea. you've got my kudos