python 3.x - How should Azure ItemPaged iterator work?

Question

I am trying to retrieve all paths from ADLS Gen2 Storage using below code:

file_system_client = service_client.get_file_system_client(file_system="my-file-system")
paths = file_system_client.get_paths()
pathlist = []
for path in paths:
    pathlist.append(path.name)

The length of pathlist is 5000. According to documentation - it is default max_results for page, as the output of get_pages() is the ItemPaged[PathProperties].

Now I can't understand how to handle this output type to get all the paths from my file system...

I also tried to iterate pages using by_pages() method, but till got only one page and then End of paging:

page_iterator = paths.by_pages()
page_iterator.next()
page iterator.current_page

[list-of-PathProperties] - 5000 items

page_iterator.next()

StopIteration: End of paging

And I definitely know that there are more paths to be retrieved from the container.

Could you help me to handle this procedure correctly?

Thanks!

question from:https://stackoverflow.com/questions/65938321/how-should-azure-itempaged-iterator-work

Answer 1

There is two ways to iterate:

• for path in file_system_client.get_paths(): will iterate all paths, not notion of pages
• for page in file_system_client.get_paths().by_pages(): will iterate on the pages, that contains paths

This means the first one is returning an iterator of path

pathlist = []
for path in paths:  # Should iterate ALL
    pathlist.append(path.name)

While the second will iterate pages of paths, therefore you would need two loops. This one is useful if you build a webpage for instance, and you need result by page (like a Google/Bing result, etc.)

pathlist = []
for page in file_system_client.get_paths().by_pages():
    for path in page:
        pathlist.append(path.name)

ItemPaged is an iterator, meaning you can consume it with everything that takes an iterator in input. You don't need a for at all

pathlist = list(file_system_client.get_paths())  # list() consumes the iterator to a list

This is for the general behavior of those classes.

Now I understand from your post that you would expect to get more than 5000 paths, which I assume means you know you have more than that in your account. If that's indeed the case, it deserves a bug investigation as the first syntax should return all, and not only the first page, please open an issue here: https://github.com/Azure/azure-sdk-for-python/issues

(I work at MS in the Azure Python SDK team)