How to download more than 5000 records from Dynamics web api using python?

Question

I'm trying to get all the records from a query to the Dynamics API, but I don't know how to break the default limit of 5000.

Currently my code looks like this:

    LOGIN_URL = 'https://login.microsoftonline.com/xxxxxxxxxxxxxxxxxxxx/oauth2/token'
    TENANT_ID = 'xxxxxxxxxxxxxxxxxxxxxx'
    CLIENT_ID = 'xxxxxxxxxxxxxxxxxxxxxx'
    GRANT_TYPE = 'client_credentials'
    CLIENT_SECRET = 'xxxxxxxxxxxxxxxxxxxxxx'
    RESOURCE = 'https://xxx.crmX.dynamics.com'
    API_ACCOUNTS = "https://xxx.crmX.dynamics.com/api/data/v9.1/accounts

    
    response = requests.post(LOGIN_URL, data={'tenant_id': TENANT_ID,
                                              'client_id': CLIENT_ID,
                                              'client_secret': CLIENT_SECRET,
                                              'grant_type': GRANT_TYPE,
                                              'resource': RESOURCE}).json()
    
    try:
        accesstoken = response['access_token']    
    
    except(KeyError):
        print('Could not get access token')
    
    customer_data = requests.get(API_ACCOUNTS, headers={
                                 'Authorization': 'Bearer ' + accesstoken})
    
    jsonData = customer_data.json()
    
    jsonD = json.dumps(jsonData)
    f = open("accounts.json", "w")
    f.write(jsonD)
    f.close()

But this way I only get the first 5000 records.Does anyone know what modifications I should make? Thank you!

question from:https://stackoverflow.com/questions/65908020/how-to-download-more-than-5000-records-from-dynamics-web-api-using-python

Answer 1

To break this barrier, you have to use paging cookie, odata.maxpagesize and @odata.nextLink. Read more

To retrieve page 2, use a GET request with the value of the @odata.nextLink property.

Sample:

"@odata.nextLink":"https://[Organization URI]/api/data/v9.0/contacts?$select=fullname,jobtitle,annualincome&$filter=contains(fullname,'(sample)')&$count=true&$skiptoken=%3Ccookie%20pagenumber=%222%22%20pagingcookie=%22%253ccookie%2520page%253d%25221%2522%253e%253ccontactid%2520last%253d%2522%257bC748FDEE-C143-E611-80D5-00155DA84802%257d%2522%2520first%253d%2522%257bB848FDEE-C143-E611-80D5-00155DA84802%257d%2522%2520%252f%253e%253c%252fcookie%253e%22%20istracking=%22False%22%20/%3E"