python - Pandas: ignore all lines following a specific string when reading a file into a DataFrame

Question

I have a pandas DataFrame which can be summarized as this:

[Header]
Some_info = some_info
[Data]
Col1    Col2
0.532   Point
0.234   Point
0.123   Point
1.455   Square
14.64   Square
[Other data]
Other1  Other2
Test1   PASS
Test2   FAIL

My goal is to read only the portion of text between [Data] and [Other data], which is variable (different length). The header has always the same length, so skiprows from pandas.read_csv can be used. However, skipfooter needs the number of lines to skip, which can change between files.

What would be the best solution here? I would like to avoid altering the file externally unless there's no other solution.

Answer 1

Numpy's genfromtxt has the ability to take a generator as an input (rather than a file directly) -- the generator can just stop yielding as soon as it hits your footer. The resulting structured array could be converted to a pandas DataFrame. It's not ideal, but it didn't look like pandas' read_csv could take the generator directly.

import numpy as np
import pandas as pd

def skip_variable_footer(infile):
    for line in infile:
        if line.startswith('[Other data]'):
            raise StopIteration
        else:
            yield line


with open(filename, 'r') as infile:
    data = np.genfromtxt(skip_variable_footer(infile), delimiter=',', names=True, dtype=None)

df = pd.DataFrame(data)