python - Efficient way for appending numpy array

Question

I will keep it simple.I have a loop that appends new row to a numpy array...what is the efficient way to do this.

n=np.zeros([1,2])
for x in [[2,3],[4,5],[7,6]]
      n=np.append(n,x,axis=1)

Now the thing is there is a [0,0] sticking to it so I have to remove it by

   del n[0]

Which seems dumb...So please tell me an efficient way to do this.

   n=np.empty([1,2])

is even worse it creates an uninitialised value.

Answer 1

A bit of technical explanation for the "why lists" part.

Internally, the problem for a list of unknown length is that it needs to fit in memory somehow regardless of its length. There are essentially two different possibilities:

1. Use a data structure (linked list, some tree structure, etc.) which makes it possible to allocate memory separately for each new element in a list.

2. Store the data in a contiguous memory area. This area has to be allocated when the list is created, and it has to be larger than what we initially need. If we get more stuff into the list, we need to try to allocate more memory, preferably at the same location. If we cannot do it at the same location, we need to allocate a bigger block and move all data.

The first approach enables all sorts of fancy insertion and deletion options, sorting, etc. However, it is slower in sequential reading and allocates more memory. Python actually uses the method #2, the lists are stored as "dynamic arrays". For more information on this, please see:

Size of list in memory

What this means is that lists are designed to be very efficient with the use of append. There is very little you can do to speed things up if you do not know the size of the list beforehand.

---

If you know even the maximum size of the list beforehand, you are probably best off allocating a numpy.array using numpy.empty (not numpy.zeros) with the maximum size and then use ndarray.resize to shrink the array once you have filled in all data.

For some reason numpy.array(l) where l is a list is often slow with large lists, whereas copying even large arrays is quite fast (I just tried to create a copy of a 100 000 000 element array; it took less than 0.5 seconds).

This discussion has more benchmarking on different options:

Fastest way to grow a numpy numeric array

I have not benchmarked the numpy.empty + ndarray.resize combo, but both should be rather microsecond than millisecond operations.