python - How can I preserve <br> as newlines with lxml.html text_content() or equivalent?

Question

I want to preserve <br> tags as when extracting the text content from lxml elements.

Example code:

fragment = '<div>This is a text node.<br/>This is another text node.<br/><br/><span>And a child element.</span><span>Another child,<br> with two text nodes</span></div>'

h = lxml.html.fromstring(fragment)

Output:

> h.text_content()
'This is a text node.This is another text node.And a child element.Another child, with two text nodes'

Answer 1

Prepending an character to the tail of each <br /> element should give the result you're expecting:

>>> import lxml.html as html
>>> fragment = '<div>This is a text node.<br/>This is another text node.<br/><br/><span>And a child element.</span><span>Another child,<br> with two text nodes</span></div>'
>>> doc = html.document_fromstring(fragment)
>>> for br in doc.xpath("*//br"):
        br.tail = "
" + br.tail if br.tail else "
"

>>> doc.text_content()
'This is a text node.
This is another text node.

And a child element.Another child,
 with two text nodes'
>>> fragment
'<div>This is a text node.<br/>This is another text node.<br/><br/><span>And a child element.</span><span>Another child,<br> with two text nodes</span></div>'