python - How do I remove rows from a numpy array based on multiple conditions?

Question

I have a file with three columns and thousands of rows. I want to remove those rows, whose items in the first column are in a certain range. For example, if the data in my file is as follows:

18  6.215   0.025
19  6.203   0.025
20  6.200   0.025
21  6.205   0.025
22  6.201   0.026
23  6.197   0.026
24  6.188   0.024
25  6.187   0.023
26  6.189   0.021
27  6.188   0.020
28  6.192   0.019
29  6.185   0.020
30  6.189   0.019
31  6.191   0.018
32  6.188   0.019
33  6.187   0.019
34  6.194   0.021
35  6.192   0.024
36  6.193   0.024
37  6.187   0.026
38  6.184   0.026
39  6.183   0.027
40  6.189   0.027

I want to remove those rows, whose first item is between 20 and 25 or between 30 and 35. The expected output is thus:

18  6.215   0.025
19  6.203   0.025
26  6.189   0.021
27  6.188   0.020
28  6.192   0.019
29  6.185   0.020
36  6.193   0.024
37  6.187   0.026
38  6.184   0.026
39  6.183   0.027
40  6.189   0.027

How could I do this?

Answer 1

If you want to keep using numpy, the solution isn't hard.

data = data[np.logical_not(np.logical_and(data[:,0] > 20, data[:,0] < 25))]
data = data[np.logical_not(np.logical_and(data[:,0] > 30, data[:,0] < 35))]

Or if you want to combine it all into one statement,

data = data[
    np.logical_not(np.logical_or(
        np.logical_and(data[:,0] > 20, data[:,0] < 25),
        np.logical_and(data[:,0] > 30, data[:,0] < 35)
    ))
]

To explain, conditional statements like data[:,0] < 25 create boolean arrays that track, element-by-element, where the condition in an array is true or false. In this case, it tells you where the first column of data is less than 25.

You can also index numpy arrays with these boolean arrays. A statement like data[data[:,0] > 30] extracts all the rows where data[:,0] > 30 is true, or all the rows where the first element is greater than 30. This kind of conditional indexing is how you extract the rows (or columns, or elements) that you want.

Finally, we need logical tools to combine boolean arrays element-by-element. Regular and, or, and not statements don't work because they try to combine the boolean arrays together as a whole. Fortunately, numpy provides a set of these tools for use in the form of np.logical_and, np.logical_or, and np.logical_not. With these, we can combine our boolean arrays element-wise to find rows that satisfy more complicated conditions.