[search api] error when query artists of a certain genre over 1000+

[irisfee]

Hello!

I have got an error when querying the artists info of a given genre. The error only occurs when the total artists of that genre is greater than 1000.

genre = 'j-pop'
artist=[]
# query the total number of artists
at=sp.search(q='genre:'+genre,limit=50,type='artist')['artists']
print('Total number of artists for '+genre+': '+str(at['total']))
# query all artists info
if at['total']>50:
for i in range(0,at['total'],50):
artist+=sp.search(q='genre:'+genre,limit=50,type='artist',offset=i)['artists']['items']

and here is the error

HTTP Error for GET to https://api.spotify.com/v1/search returned 404 due to Not found.
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/spotipy/client.py in _internal_call(self, method, url, payload, params)
    244 
--> 245             response.raise_for_status()
    246             results = response.json()

~/anaconda3/lib/python3.6/site-packages/requests/models.py in raise_for_status(self)
    940         if http_error_msg:
--> 941             raise HTTPError(http_error_msg, response=self)
    942 

HTTPError: 404 Client Error: Not Found for url: https://api.spotify.com/v1/search?q=genre%3Aj-pop&limit=50&offset=1000&type=artist

During handling of the above exception, another exception occurred:

SpotifyException                          Traceback (most recent call last)
<ipython-input-124-a23c0e04674c> in <module>
      6 if at['total']>50:
      7     for i in range(0,at['total'],50):
----> 8         artist+=sp.search(q='genre:'+genre,limit=50,type='artist',offset=i)['artists']['items']
      9 # save_json("data/artists/{}_artists_info".format(genre),artist)

~/anaconda3/lib/python3.6/site-packages/spotipy/client.py in search(self, q, limit, offset, type, market)
    546         """
    547         return self._get(
--> 548             "search", q=q, limit=limit, offset=offset, type=type, market=market
    549         )
    550 

~/anaconda3/lib/python3.6/site-packages/spotipy/client.py in _get(self, url, args, payload, **kwargs)
    289             kwargs.update(args)
    290 
--> 291         return self._internal_call("GET", url, payload, kwargs)
    292 
    293     def _post(self, url, args=None, payload=None, **kwargs):

~/anaconda3/lib/python3.6/site-packages/spotipy/client.py in _internal_call(self, method, url, payload, params)
    264                 "%s:\n %s" % (response.url, msg),
    265                 reason=reason,
--> 266                 headers=response.headers,
    267             )
    268         except requests.exceptions.RetryError as retry_error:

SpotifyException: http status: 404, code:-1 - https://api.spotify.com/v1/search?q=genre%3Aj-pop&amp;limit=50&amp;offset=1000&amp;type=artist:
 Not found., reason: None

The query works well when the total artists is smaller than 1000. For example, when i query information about mandopop, no error occurs.

I wonder if there is a way to solve this.

Thanks 

[Peter_Schorn]

The documentation for the search endpoint clearly says "Maximum offset (including limit): 1,000."

[hendrywijaya98]

then, is there any technique, to get song data with a maximum offset of more than 1000?