Searchable/copiable text in PDFs

[kyrias]

I've encountered something that to me feels a bit strange, and I'm not quite sure what the difference between what XeLaTeX and SILE does is in this case.

The PDF generated from the following SIL document, enabling the ss02 font feature which causes the text to be rendered with v in word-initial positions and u otherwise, ends up without searchable/copiable text in MuPDF and Firefox's PDF.js. Copying the text leads to copying 0xAD3/0xAD4/0xAD5 which are the font's substituted glyphs. Running mutool draw -F text -o - on the file results in ��u�u, with the substituted glyphs being replaced with replacement characters.

\begin[class=book]{document}
\font[size=12pt,filename="EBGaramond12-Regular.otf",features="+ss02"]
uvuvu
\end{document}

Meanwhile, if I run the following document through XeLaTeX then the resulting PDF file is searchable and has copiable text. Running it through the same mutool command results in uvuvu.

\documentclass[12pt]{article}
\usepackage{fontspec}
\begin{document}
\fontspec {EBGaramond12-Regular.otf}[RawFeature=+ss02]
uvuvu
\end{document}

I've tried to the best of my ability to inspect the generated PDF files to see what might differ between them, but I'm not very familiar with how the format works and so I'm a bit stuck.

While looking around for ways to make it work I ran into #494 (comment), but it doesn't seem to make a difference.

I would appreciate any help in figuring out how the files differ, and any hints for how to make the SILE-generated PDF functionally equivalent to the XeLaTeX-generated one.

sile.pdf
xelatex.pdf

[Omikhleia]

First, the sile.pdf file you attached shows "uvuvu", not "vuuuu" as in your XeLaTeX one -- and upon inspection, it's not using the EB Garamond font but the Gentium Plus font:

mutool info sile.pdf 
sile.pdf:

PDF-1.5
Info object (2 0 R):
<</Producer(SILE v0.14.13 \(LuaJIT 2.1.1700008891\))/CreationDate(D:20231210195817+01'00')>>
Pages: 1

Retrieving info from pages 1-1...
Fonts (1):
	1	(3 0 R):	Type0 'JANDVM+GentiumPlus' Identity-H (5 0 R)

So you might have had another issue here, or didn't provide the correct PDF.

Doing it on my own system with EB Garamond installed:

\begin[class=book]{document}
\font[size=12pt,family=EB Garamond,features="+ss02"]
uvuvu
\end{document}

Then I got "vuuuu" with the feature applied, and the file correctly reports the correct font.

I can try to investigate from this point:

• On your XeLaTeX file:

  $ mutool trace xelatex.pdf 
  <document filename="xelatex.pdf">
  <page number="1" mediabox="0 0 595.28 841.89">
  <set_default_colorspaces gray="DeviceGray" rgb="DeviceRGB" cmyk="DeviceCMYK" oi="None"/>
  <fill_text colorspace="DeviceGray" color="0" transform="1 0 0 -1 72 72">
      <span font="CGUAVJ+EBGaramond12-Regular" wmode="0" bidi="0" trm="11.9552 0 0 11.9552">
          <g unicode="u" glyph="2772" x="56.787" y="-65.753" />
          <g unicode="v" glyph="2773" x="62.023378" y="-65.753" />
          <g unicode="u" glyph="86" x="68.32377" y="-65.753" />
          <g unicode="v" glyph="2773" x="74.62416" y="-65.753" />
          <g unicode="u" glyph="86" x="80.92455" y="-65.753" />
      </span>
      <span font="EIAGNJ+LMRoman12-Regular" wmode="0" bidi="0" trm="11.9552 0 0 11.9552">
          <g unicode="1" glyph="82" x="230.197" y="-630.13699" />
      </span>
  </fill_text>
  </page>
  </document>

• On my re-generated SILE file:

  $ mutool trace sile.pdf 
  <document filename="sile.pdf">
  <page number="1" mediabox="0 0 595.28 841.89">
  <set_default_colorspaces gray="DeviceGray" rgb="DeviceRGB" cmyk="DeviceCMYK" oi="None"/>
  <fill_text colorspace="DeviceGray" color="0" transform="1 0 0 -1 0 0">
      <span font="ZZYWLV+EBGaramond12-Regular" wmode="0" bidi="0" trm="12 0 0 12">
          <g unicode="U+fffd" glyph="2923" x="69.41" y="-102.64" />
          <g unicode="U+fffd" glyph="2924" x="74.666" y="-102.64" />
          <g unicode="u" glyph="86" x="80.99" y="-102.64" />
          <g unicode="U+fffd" glyph="2924" x="87.313999" y="-102.64" />
          <g unicode="u" glyph="86" x="93.63799" y="-102.64" />
      </span>
  </fill_text>
  <fill_text colorspace="DeviceGray" color="0" transform="1 0 0 -1 0 0">
      <span font="ZZYWLV+EBGaramond12-Regular" wmode="0" bidi="0" trm="12 0 0 12">
        <g unicode="1" glyph="18" x="278.94" y="-731.42" />
      </span>
  </fill_text>
  </page>
  </document>

The glyph numbers, which I assume to be the GID in the embedded font, differ -- but I'd assume (not knowingly) that this might be due to different versions of the font, or to the font embedding/subsetting perhaps... The coordinates also differ, but that's rather logical (different page layouts), as well as the folio numbering (and the font it uses)

Then the remaining notable difference is the unicode equivalent value of the glyphs.

• In the XeLaTeX case, they map to existing unicode values.
• In the SILE case, they map to U+FFFD (replacement character) -- This might explain the inability to search/copy them, although I've no idea about the root cause. Our libtexpdf outputter is based on a fairly old version of xdvipfmx (your XeLaTeX example reports using version 20220710, likely more recent that the library @simoncozens derived from it); that might be the reason... or not...

[Omikhleia]

This being said, the zero-with hboxes introduced by pdf:literal might not be cool in all workflows, so this kind of "override" is perhaps not perfect...

[Omikhleia]

Another hack avoiding the pdf:literal and its boxes...

\begin[class=book]{document}
\lua{
if SILE.outputter._name == "libtexpdf" then
  local pdf = require("justenoughlibtexpdf")
  local savedOriginal = SILE.outputter.drawHbox
  SILE.outputter.drawHbox = function (self, value, width)
    self:_ensureInit()
    if not value.glyphString then return end
    pdf.add_content("/Span << /ActualText (" .. value.text .. ") >>\nBDC\n")
    savedOriginal(self, value, width)
    pdf.add_content("\nEMC")
  end
end
}
\font[size=12pt,family=EB Garamond,features="+ss02"]
uvuvu
\end{document}

[Omikhleia]

And of course, testing on a French text with accents... I had forgotten that the node's text is in Unicode, so the above should be fixed as follows:

    local txt = SU.utf8_to_utf16be_hexencoded(value.text)
    pdf.add_content("/Span << /ActualText <" .. txt .. "> >>\nBDC\n")

And similarly for thesetpar example implementation.

(Here again I wonder if the pdfstructure package would do things properly or not...)

[Omikhleia]

Cross-linking #494 and #110

[kyrias]

First, the sile.pdf file you attached shows "uvuvu", not "vuuuu" as in your XeLaTeX one -- and upon inspection, it's not using the EB Garamond font but the Gentium Plus font:

Ah, silly, I attached the wrong file. 😅

---

Anyway, thanks, this should hopefully help me track down what needs to be changed to make it map the glyphs back to the correct Unicode code points.