Too Many Requests (429 Error) - blocked for 13 hours

[garou-garou]

Plan

Premium Family

Country

United States

Device

Macbook Pro M1 2020

Operating System

MacOS 13.3.1

My Question or Issue

I am running a Python script I wrote that uses the spotipy library to make Web API requests. My script will make a single request to search for a track, then wait 1 second, then make another request. I am trying to do this for about 34,000 tracks. After about 24,000 requests over the course of about 12 hours, my script stopped running. I then made a manual API request with a curl command, and saw that the response stated "Too many requests", with `retry-after: 39437` in the header (asking to retry after about 13 hours).

The only documentation I can find about rate limits suggests that Spotify rate limits are calculated on a rolling 30 second basis. I have no idea why my script was succeeding for 12 hours straight, only to suddenly be blocked, unless there's also some daily limit in place.

What is the actual rate limit policy? I need to know so that I can avoid hitting the rate limit again in the future.

[teed_breezy]

I'm having a similar issue; I'm currently working on a personal project using the spotipy library in Python to make requests to the audio_features endpoint. My script processes approximately 2500 tracks, with a 1-second delay between each call. After about 1900 requests, I started receiving HTTP 429 errors. This wasn't entirely unexpected, but I noticed that these errors don't come with a Retry-After header. Like garou-garou, I have now been unable to effectively use the API for over 12 hours.

I have a couple of questions:

• Does the audio_features endpoint have a different rate limiting policy compared to other endpoints?
• Are there known issues or best practices for handling 429 errors specifically for this endpoint, especially given the absence of the Retry-After header?

Any advice on how to handle this situation would be greatly appreciated. I'm flexible with spreading out my requests over time if necessary.

Thanks in advance for your assistance.

[Ximzend]

@teed_breezy, according to the documentation of Spotify you can provide a list of track URIs, URLs or IDs, maximum: 100 ids. So, 2500 tracks should only take 25 request; not (more than) 1900.

How you can split a list in chunks, you can read here.

[teed_breezy]

Hi Ximzend, thanks for your reply. So as I got the audio features for about 1900 tracks, I guess I made 19 requests before getting rate limited? I am passing a list of track ids to my function. The spotipy docs say ‘maximum: 100 ids’. Maybe my list is being split into chunks already? Thanks again.

Thanks again.

[Ximzend]

Hi teed_breezy, can you please (a part of) your code? On desktop, you can it as a code block.

[teed_breezy]

Hi Ximzend - sure.

It’s just ‘tempo’ that I’m trying to get for a list of tracks. It works, but I get rate limited before the list completes, and I’m still get a 429 if I run this code, almost 24 hours after I first got one. I see on my dashboard I made just under 2k requests to /v1/audio-features. I don’t know how long it will be before I can make another request.

Anyway, I appreciate your help. Here is my function:

def get_tempo(long_track_id_lst):
    tempo_lst = []

    for track_id in long_track_id_lst:
        try:
            af = sp.audio_features(track_id)
            
            if af and af[0]:
                tempo = af[0]['tempo']
                tempo_lst.append(tempo)
            else:
                tempo_lst.append(0.0)
            
            time.sleep(1)

        except requests.exceptions.HTTPError as e:
            if e.response.status_code == 429:
                retry_after = int(e.response.headers.get('Retry-After', 1))
                print(f"Rate limited. Retrying after {retry_after} seconds.")
                time.sleep(retry_after + 1) 
                continue
            else:
                raise
        except Exception as e:
            print(f"Failed to get audio features for track ID {track_id}: {e}")
            tempo_lst.append(0.0)

    return tempo_lst

[Ximzend]

Here is modified code that uses chunks:

def get_tempo(long_track_id_lst):
    chunk_size = 100
    tempo_lst = []

    # Divide the track IDs into chunks
    chunks = [long_track_id_lst[i:i + chunk_size] for i in range(0, len(long_track_id_lst), chunk_size)]

    for chunk in chunks:
        try:
            # Get audio features for the chunk of track IDs
            audio_features = sp.audio_features(chunk)

            for af in audio_features:
                if af and af[0]:
                    tempo = af[0]['tempo']
                    tempo_lst.append(tempo)
                else:
                    tempo_lst.append(0.0)

            time.sleep(1)

        except requests.exceptions.HTTPError as e:
            if e.response.status_code == 429:
                retry_after = int(e.response.headers.get('Retry-After', 1))
                print(f"Rate limited. Retrying after {retry_after} seconds.")
                time.sleep(retry_after + 1)
                continue
            else:
                raise
        except Exception as e:
            print(f"Failed to get audio features for some track IDs: {e}")
            tempo_lst.extend([0.0] * len(chunk))

    return tempo_lst

I don't know when and if the rate limit gets lifted, and if you are also rate limited when you create another application.
Make sure you use the modified code (chunks of 100 ids), so you don't make 100 times more requests than needed.

[teed_breezy]

Nice - that is appreciated. I'll give it a try when (and if!) I can access the endpoint again. Thanks.

[bjayesh47]

Hi, what were your findings regarding rate limiting and batch processing?

[teed_breezy]

The above issue turned out to be a rate limiting one. I found that after I’d backed off the endpoint for a few days I could access it again. I did that using the batch processing method described above and have not been rate limited since. Having said that, I am more conscious of it now and tend to split my requests over long time periods.

[jspayton15]

I am getting "too many 429 error responses" and don't get the retry-after header either. I haven't tried again as I feel it restarts the timer. Is there a Spotify tech who can advise here?