pdfminer - PDF text extraction returns wrong characters due to ToUnicode map

Question

I am trying to extract text from a foreign language PDF file using PDFMiner, but am being foiled by a ToUnicode statement. The file behaves strangely even under normal PDF viewers.

For example, here is a screenshot from some text in the file:

But if I select and copy the text, it looks like this:

?????

You can see several characters have changed, in particular the second-to-last character.

Not surprisingly, PDFMiner extracts the incorrect text. But every PDF viewer manages to display these data correctly. I suspect the issue is either the ToUnicode map, or something with conjoined characters. The desired letter should be a sequence of 0x915, 0x94D, 0x937. PDFMiner only reports 0x915, which describes a different character.

What do I need to do to get PDFMiner to extract text correctly, i.e. as in the image rather than the copy-pasted text?

Here is a link to the PDF in question.

Answer 1

In short:

Your PDF does not contain the information required for correct text extraction without the use of OCR.

In detail:

Both the ToUnicode Map and the Unicode entries in the font program of the embedded subset of Mangal-Regular in your PDF claim that these four glyphs

all represent the same Unicode code point, 0x915.

Thus, any text extraction program which does not look at the drawn glyph (i.e. not attempt OCR) will return 0x915 for either one of those glyphs.

Background:

You seem to wonder why the PDF viewers correctly display the text but text extraction (copy&paste or PDFMiner) does not correctly extract.

The reason is that PDF as a format does not contain the text as such. It contains pointers (direct ones or via mappings) to glyph drawing instructions in embedded font programs. Using these pointers the PDF is drawn as you expect.

Furthermore it can contain extra information mapping such glyph pointers to Unicode code points. Such extra information is used by text extracting programs. In case of your PDF these mappings are incorrect and, therefore, extracted text is incorrect.