python - Elegant way to skip elements in an iterable

Question

I've got a large iterable, in fact, a large iterable given by:

itertools.permutations(range(10))

I would like to access to the millionth element. I alredy have problem solved in some different ways.

1. Casting iterable to list and getting 1000000th element:

   return list(permutations(range(10)))[999999]
   

2. Manually skiping elements till 999999:

   p = permutations(range(10))
   for i in xrange(999999): p.next()
   return p.next()
   

3. Manually skiping elements v2:

   p = permutations(range(10))
   for i, element in enumerate(p):
       if i == 999999:
           return element
   

4. Using islice from itertools:

   return islice(permutations(range(10)), 999999, 1000000).next()
   

But I still don't feel like none of them is the python's elegant way to do that. First option is just too expensive, it needs to compute the whole iterable just to access a single element. If I'm not wrong, islice does internally the same computation I just did in method 2, and is almost exactly as 3rd, maybe it has even more redundant operations.

So, I'm just curious, wondering if there is in python some other way to access to a concrete element of an iterable, or at least to skip the first elements, in some more elegant way, or if I just need to use one of the aboves.

Answer 1

Use the itertools recipe consume to skip n elements:

def consume(iterator, n):
    "Advance the iterator n-steps ahead. If n is none, consume entirely."
    # Use functions that consume iterators at C speed.
    if n is None:
        # feed the entire iterator into a zero-length deque
        collections.deque(iterator, maxlen=0)
    else:
        # advance to the empty slice starting at position n
        next(islice(iterator, n, n), None)

Note the islice() call there; it uses n, n, effectively not returning anything, and the next() function falls back to the default.

Simplified to your example, where you want to skip 999999 elements, then return element 1000000:

return next(islice(permutations(range(10)), 999999, 1000000))

islice() processes the iterator in C, something that Python loops cannot beat.

To illustrate, here are the timings for just 10 repeats of each method:

>>> from itertools import islice, permutations
>>> from timeit import timeit
>>> def list_index():
...     return list(permutations(range(10)))[999999]
... 
>>> def for_loop():
...     p = permutations(range(10))
...     for i in xrange(999999): p.next()
...     return p.next()
... 
>>> def enumerate_loop():
...     p = permutations(range(10))
...     for i, element in enumerate(p):
...         if i == 999999:
...             return element
... 
>>> def islice_next():
...     return next(islice(permutations(range(10)), 999999, 1000000))
... 
>>> timeit('f()', 'from __main__ import list_index as f', number=10)
5.550895929336548
>>> timeit('f()', 'from __main__ import for_loop as f', number=10)
1.6166789531707764
>>> timeit('f()', 'from __main__ import enumerate_loop as f', number=10)
1.2498459815979004
>>> timeit('f()', 'from __main__ import islice_next as f', number=10)
0.18969106674194336

The islice() method is nearly 7 times faster than the next fastest method.