python - Numpy reshape on view

Question

I'm confused about the results of numpy reshape operated on a view. In the following q.flags shows that it does not own the data, but q.base is neither x nor y, so what is it? I'm surprised to see that q.strides is 8 which means that it gets the next element by every time move 8 bytes in memory (if I understand correctly). However if none of the arrays other than x owns data, the only data buffer is from x, which does not permit getting the next element of q by moving 8 bytes.

In [99]: x = np.random.rand(4, 4)

In [100]: y = x.T

In [101]: q = y.reshape(16)

In [102]: q.base is y
Out[102]: False

In [103]: q.base is x
Out[103]: False

In [104]: y.flags
Out[104]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [105]: q.flags
Out[105]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [106]: q.strides
Out[106]: (8,)

In [107]: x
Out[107]: 
array([[ 0.62529694,  0.20813211,  0.73932923,  0.43183722],
       [ 0.09755023,  0.67082005,  0.78412615,  0.40307291],
       [ 0.2138691 ,  0.35191283,  0.57455781,  0.2449898 ],
       [ 0.36476299,  0.36590522,  0.24371933,  0.24837697]])

In [108]: q
Out[108]: 
array([ 0.62529694,  0.09755023,  0.2138691 ,  0.36476299,  0.20813211,
        0.67082005,  0.35191283,  0.36590522,  0.73932923,  0.78412615,
        0.57455781,  0.24371933,  0.43183722,  0.40307291,  0.2449898 ,
        0.24837697])

UPDATE:

It turns out that this question has been asked in the numpy discussion forum: http://numpy-discussion.10968.n7.nabble.com/OWNDATA-flag-and-reshape-views-vs-copies-td10363.html

Answer 1

In short: you cannot always rely on the ndarray.flags['OWNDATA'].

>>> import numpy as np
>>> x = np.random.rand(2,2)
>>> y = x.T
>>> q = y.reshape(4)
>>> y[0,0]
0.86751629121019136
>>> y[0,0] = 1
>>> q
array([ 0.86751629,  0.87671107,  0.65239976,  0.41761267])
>>> x
array([[ 1.        ,  0.65239976],
       [ 0.87671107,  0.41761267]])
>>> y
array([[ 1.        ,  0.87671107],
       [ 0.65239976,  0.41761267]])
>>> y.flags['OWNDATA']
False
>>> x.flags['OWNDATA']
True
>>> q.flags['OWNDATA']
False
>>> np.may_share_memory(x,y)
True
>>> np.may_share_memory(x,q)
False

Because q didn't reflect the change in the first element, like x or y, it must somehow be the owner of the data (somehow is explained below).

There is more discussion about the OWNDATA flag over at the numpy-discussion mailinglist. In the How can I tell if NumPy creates a view or a copy? question, it is briefly mentioned that simply checking the flags.owndata of an ndarray sometimes seems to fail and that it seems unreliable, as you mention. That's because every ndarray also has a base attribute:

the base of an ndarray is a reference to another array if the memory originated elsewhere (otherwise, the base is None). The operation y.reshape(4) creates a copy, not a view, because the strides of y are (8,16). To get it reshaped (C-contiguous) to (4,), the memory pointer would have to jump 0->16->8->24, which is not doable with a single stride. Thus q.base points to the memory location generated by the forced-copy-operation y.reshape, which has the same shape as y, but copied elements and thus has normal strides again: (16, 8). q.base is thus not bound to by any other name as it was the result of the forced-copy operation y.reshape(4). Only now can the object q.base be viewed in a (4,) shape, because the strides allow this. q is then indeed a view on q.base.

For most people it would be confusing to see that q.flags.owndata is False, because, as shown above, it is not a view on y. However, it is a view on a copy of y. That copy, q.base, is the owner of the data however. Thus the flags are actually correct, if you inspect closely.