python - Parsing a table with rowspan and colspan

Question

I have a table that I need to parse, specifically it is a school schedule with 4 blocks of time, and 5 blocks of days for every week. I've attempted to parse it, but honestly have not gotten very far because I am stuck with how to deal with rowspan and colspan attributes, because they essentially mean there is a lack of data that I need to continue.

As an example of what I want to do, here's a table:

<tr>
    <td colspan="2" rowspan="4">#1</td>
    <td rowspan="4">#2</td>
    <td rowspan="2">#3</td>
    <td rowspan="2">#4</td>
</tr>

<tr>
</tr>

<tr>
    <td rowspan="2">#5</td>
    <td rowspan="2">#6</td>
</tr>

<tr>
</tr>

I want to take that table and convert it into this list:

[[1,1,2,3,4],
 [1,1,2,3,4],
 [1,1,2,5,6],
 [1,1,2,5,6]]

Right now I'm getting a flat list, similar to this:

[1,2,3,4,5,6]

But in dictionary form, with information regarding how many columns and rows it spans, a description of it and what week it's in.

Obviously this needs to work for every possibility of rowspan/colspan, and for multiple weeks in the same table.

The html is not as clean as I've portrayed it, there are a lot of attributes I've left out, and the text is obviously not as clean cut as 1,2,3,4 but rather blocks of descriptive text. But if I could get this part resolved then it should be easy enough to incorporate into what I've already written.

I've been using lxml.html and Python to do this, but I'm open to using other modules if it provides an easier solution.

I hope someone can help me, because I really don't know what to do.

EDIT:

<table>
    <tr>
        <td> </td>
        <td> </td>
        <td> </td>
        <td rowspan="4">Thing</td>
        <td> </td>
    </tr>
    <tr>
        <td> </td>
        <td> </td>
        <td> </td>
        <td> </td>
    </tr>
    <tr>
        <td> </td>
        <td> </td>
        <td> </td>
        <td> </td>
    </tr>
    <tr>
        <td> </td>
        <td> </td>
        <td> </td>
        <td> </td>
    </tr>
</table>

This is causing me some problems, this is outputting

[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', ' ', ' ']
[' ', ' ', ' ', ' ', ' ']
[' ', ' ', ' ', ' ', ' ']

With the code provided by reclosedev, what do I need to change to adapt it so it outputs

[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', 'Thing', ' ']
[' ', ' ', ' ', 'Thing', ' ']

Instead?

EDIT2: Using reclosedev's new function, it's approaching a solution, but there are still cases where it fails to place cells correctly:

<table> 
    <tr>
        <td> </td>
        <td rowspan="2"> DMAT Aud. 6 </td>
        <td rowspan="4"> Exam</td>
        <td rowspan="2"> DMAT Aud. 7</td>
        <td> </td>
    </tr>
    <tr>
        <td> </td>
        <td rowspan="2"> CART Aud. 4</td>
    </tr>
    <tr>
        <td> </td>
        <td rowspan="2"> CART Aud. 4</td>
        <td rowspan="2"> OOP Aud. 7</td>
    </tr>
    <tr>
        <td> </td>
        <td> </td>
    </tr>
</table> 

With this, the original table shows it as such:

[
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' '],
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' CART Aud. 4'],
[' ', ' CART Aud. 4' , ' Exam', ' OOP Aud. 7' , ' CART Aud. 4'],
[' ', ' CART Aud. 4' , ' Exam', ' OOP Aud. 7' , ' ']
]

But the new call outputs this:

[
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' '],
[' ', ' DMAT Aud. 6 ', ' Exam', ' DMAT Aud. 7', ' CART Aud. 4'],
[' ', ' CART Aud. 4' , ' Exam', ' CART Aud. 4', ' OOP Aud. 7'],
[' ', ' CART Aud. 4' , ' Exam', ' OOP Aud. 7' , ' ']
]

Answer 1

UPDATE (removed previous function)

UPDATE2 fixed and simplified.

My first function was wrong. Here's another one, it's working but needs tests:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from collections import defaultdict


def table_to_list(table):
    dct = table_to_2d_dict(table)
    return list(iter_2d_dict(dct))


def table_to_2d_dict(table):
    result = defaultdict(lambda : defaultdict(unicode))
    for row_i, row in enumerate(table.xpath('./tr')):
        for col_i, col in enumerate(row.xpath('./td|./th')):
            colspan = int(col.get('colspan', 1))
            rowspan = int(col.get('rowspan', 1))
            col_data = col.text_content()
            while row_i in result and col_i in result[row_i]:
                col_i += 1
            for i in range(row_i, row_i + rowspan):
                for j in range(col_i, col_i + colspan):
                    result[i][j] = col_data
    return result


def iter_2d_dict(dct):
    for i, row in sorted(dct.items()):
        cols = []
        for j, col in sorted(row.items()):
            cols.append(col)
        yield cols


if __name__ == '__main__':
    import lxml.html
    from pprint import pprint

    doc = lxml.html.parse('tables.html')
    for table_el in doc.xpath('//table'):
        table = table_to_list(table_el)
        pprint(table)

tables.html:

<table border="1">
    <tr>
        <td>1 </td>
        <td>1 </td>
        <td>1 </td>
        <td rowspan="4">Thing</td>
        <td>1 </td>
    </tr>
    <tr>
        <td>2 </td>
        <td>2 </td>
        <td>2 </td>
        <td>2 </td>
    </tr>
    <tr>
        <td>3 </td>
        <td>3 </td>
        <td>3 </td>
        <td>3 </td>
    </tr>
    <tr>
        <td>4 </td>
        <td>4 </td>
        <td>4 </td>
        <td>4 </td>
    </tr>
</table>

<table border="1">
<tr>
    <td colspan="2" rowspan="4">#1</td>
    <td rowspan="4">#2</td>
    <td rowspan="2">#3</td>
    <td rowspan="2">#4</td>
</tr>
<tr></tr>
<tr>
    <td rowspan="2">#5</td>
    <td rowspan="2">#6</td>
</tr>
<tr></tr>
</table>

Output:

[['1 ', '1 ', '1 ', 'Thing', '1 '],
 ['2 ', '2 ', '2 ', 'Thing', '2 '],
 ['3 ', '3 ', '3 ', 'Thing', '3 '],
 ['4 ', '4 ', '4 ', 'Thing', '4 ']]
[['#1', '#1', '#2', '#3', '#4'],
 ['#1', '#1', '#2', '#3', '#4'],
 ['#1', '#1', '#2', '#5', '#6'],
 ['#1', '#1', '#2', '#5', '#6']]