python - Why does the UnboundLocalError occur on the second variable of the flat comprehension?

Question

I answered a question here: comprehension list in python2 works fine but i get an error in python3

OP's error was using the same variables for max range and indices:

x = 12
y = 10
z = 12
n = 100

ret_list = [ (x,y,z) for x in range(x+1) for y in range(y+1) for z in range(z+1) if x+y+z!=n ]

This is a Python-3 error only, and related to the scopes that were added to the comprehension to avoid the variables defined here "leaking". Changing the variable names fixes that.

The error is:

UnboundLocalError: local variable 'y' referenced before assignment

because outer, global y is shadowed by the local scope.

My question is: why do I get the error on y and not on z or x ?

EDIT: If I remove the loop on x, the error moves to z:

>> ret_list = [ (x,y,z) for y in range(y+1) for z in range(z+1) if x+y+z!=n ]
UnboundLocalError: local variable 'z' referenced before assignment

If I just do one loop:

ret_list = [ (x,y,z) for y in range(y+1) if x+y+z!=n ]

it works. So I'm suspecting that the first range function is evaluated before all the other expressions, which leaves the value of x intact. But the exact reason is still to be found. Using Python 3.4.3.

Answer 1

This behaviour is (implicitly) described in the reference documentation (emphasis mine).

However, aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don’t “leak” into the enclosing scope.

The iterable expression in the leftmost for clause is evaluated directly in the enclosing scope and then passed as an argument to the implictly [sic] nested scope. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable. For example: [x*y for x in range(10) for y in range(x, x+10)].

This means that:

list_ = [(x, y) for x in range(x) for y in range(y)]

equivalent to:

def f(iter_):
    for x in iter_:
        for y in range(y):
            yield x, y

list_ = list(f(iter(range(x))))

As the name x in for the leftmost iterable is read in the enclosing scope as opposed to the nested scope then there is no name conflict between these two uses of x. The same is not true for y, which is why it is where the UnboundLocalError occurs.

As to why this happens: a list comprehension is more-or-less syntactic sugar for list(<generator expression>), so it's going to be using the same code path as a generator expression (or at least behave in the same way). Generator expressions evaluate the iterable expression in the leftmost for clause to make error handling when the generator expression somewhat saner. Consider the following code:

y = None                             # line 1
gen = (x + 1 for x in range(y + 1))  # line 2
item = next(gen)                     # line 3

y is clearly the wrong type and so the addition will raise a TypeError. By evaluating range(y + 1) immediately that type error is raised on line 2 rather than line 3. Thus, it is easier to diagnose where and why the problem occurred. Had it occurred on line 3 then you might mistakenly assume that it was the x + 1 statement that caused the error.

There is a bug report here that mentions this behaviour. It was resolved as "not a bug" for reason that it is desirable that list comprehensions and generator expressions have the same behaviour.