Bug#1036826: po4a: Escape sequence \c encountered. This is not completely handled yet.

[Helge Kreutzmann]

Package: po4a
Version: 0.69-1
Severity: minor

some man pages we translate have the following:

.SH OPTIONS
.I Gawk
accepts the following options.
Standard options are listed first, followed by options for
.I gawk
extensions, listed alphabetically by short option.
.TP
.BI \-f " program-file\fR,\fP "\c
.BI \-\^\-file " program-file"
Read the \*(AK program source from the file

The "\c" causes the following (fatal) warning, i.e. no output:

Escape sequence \c encountered. This is not completely handled yet.

As a workaround I'm currently considering patching the source files "on
the fly" to get the build to proceed.

Until recently I was ignoring this message, because it implies that
just some detail is not properly handled; however I did not realize that
the build fails.

(This is now urgent as some man pages build for bullseye, but no
longer for bookworm, and once bullseye translations are gone, the
entire translation is gone - no good).

So there are two solutions:
a) Reword the message to clearly state that the build is aborted
b) Ignore the escape sequence (or properly handle it) change, i.e.
actually build the pot file
In this case, the warning might remain as is.

Obviously, I would prefer b) (and potentially a sane workaround until
then).


-- System Information:
Debian Release: 12.0
APT prefers testing-security
APT policy: (500, 'testing-security'), (500, 'testing')
Architecture: amd64 (x86_64)

Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to de_DE.UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages po4a depends on:
ii gettext 0.21-12
ii libpod-parser-perl 1.65-1
ii libsgmls-perl 1.03ii-38
ii libsyntax-keyword-try-perl 0.28-1
ii libyaml-tiny-perl 1.73-1
ii opensp 1.5.2-13+b2
ii perl 5.36.0-7

Versions of packages po4a recommends:
ii liblocale-gettext-perl 1.07-5
ii libterm-readkey-perl 2.38-2+b1
ii libtext-wrapi18n-perl 0.06-10
ii libunicode-linebreak-perl 0.0.20190101-1+b5

po4a suggests no packages.

-- no debconf information

[Helge Kreutzmann]

Hello Bjarni,

Post by Helge Kreutzmann
[...]
.BI \-f " program-file\fR,\fP "\c
.BI \-\^\-file " program-file"

This is a wrong use of '\c', as its purpose is to join the output of
two macros _without_ an intervening space character.
So remove ' ' and '\c', changing
.BI \-f " program-file\fR,\fP "\c
to
.BI \-f " program-file\fR,\fP"

I'm currently looking for the best regular expressions to catch all
occurences on the fly before converting.

And I'll report this to the source package, i.e. the author of this
page. This was introduced recently, i.e. in bullseye it is not
present.

Thanks for the quick analysis.

Greetings

Helge

[Helge Kreutzmann]

Hello Bjarni,

Post by Helge Kreutzmann
[...]
.BI \-f " program-file\fR,\fP "\c
.BI \-\^\-file " program-file"

This is a wrong use of '\c', as its purpose is to join the output of
two macros _without_ an intervening space character.
So remove ' ' and '\c', changing
.BI \-f " program-file\fR,\fP "\c
to
.BI \-f " program-file\fR,\fP"

Thanks, this make the build proceed, however, now it dies in the
following line:
.TP
.BI \-F " fs\fR, \fP"\c
.BI \-\^\-field-separator " fs"

Escape sequence \c encountered. This is not completely handled yet.

Note that here there is no space before the "

Removing also this (and subsequent) "\c" makes the build proceed,
however, in other files "\c" exists as well and I'm vary of removing
the as well.

I *think* the difference is that the failing lines have a ".BI" at the
beginnig. (And the non failing do not.) Does this make sense to you?

Thanks for your support!

Greetings

Helge

[Helge Kreutzmann]

Hello Bjarni,

Post by Helge Kreutzmann
Hello Bjarni,

Post by Helge Kreutzmann
[...]
.BI \-f " program-file\fR,\fP "\c
.BI \-\^\-file " program-file"

This is a wrong use of '\c', as its purpose is to join the output of
two macros _without_ an intervening space character.
So remove ' ' and '\c', changing
.BI \-f " program-file\fR,\fP "\c
to
.BI \-f " program-file\fR,\fP"

My change is wrong as I ignored the 'TP' macro, which uses only one
(logical) line, but I made two physical lines.
The simples change is to join the two lines with an '\', and put the
space after the comma (punctuation), so

.TP
.BI something\c
.BI somethingelse

needs to become
.TP
.BI something\
somethingelse

The problem is, that I use sed to fix this (until either upstream
fixes its man page or po4a has a workaround). (And I don't care about
space etc, the translator will need to make her own anyway.)

From trial and error I noticed the the second ".BI" does not cunfuse
po4a.

So I check if

.BI something\c → .BI something\

I unique enough to only apply to the two man pages using it and does
th trick.

Thanks for your efforts!

Greetings

Helge

[Helge Kreutzmann]

severity 1036826 important
thanks

Hello Martin,
you have been quite in this discussion. \c occurs in more and more man
pages, and currently the build fails for them. In turn, they are no
longer translated.

Could you kindly check if you could add support for "\c" or is there
another workaround?

Thanks for your support!

Greetings

Helge

[Helge Kreutzmann]

Hello Martin,

Instead, I'd appreciate if you could do a merge request with a test file, along
with the expected output. It'd save me the time to dig into the discussion of
this bug. 
I'm not saying that I won't fix it w/o this test case. I'm just saying that
providing a test case is a better approach to speedup the fix than severity
abuse.

I hope explaining the test file in this bug is fine as well, because
I'm not sure what to do exactly merge and how.

The test case is groff(1) as it is in Debian unstable:

$ LC_ALL=C po4a-updatepo -f man --no-deprecation --option groff_code=verbatim --option generated --option untranslated="}1,Ds,zY,zZ,Ee,ES,dT,FN,NE,NS,EX,EE,Id,rstReportMargin,INDENT,UNINDENT,UN,a.RE,\|" --option unknown_macros=untranslated --master groff.1 -M utf-8 -p test.pot
groff.1:2279: (po4a::man)
Escape sequence \c encountered. This is not completely
handled yet.

And there is no output.

If I do a crude preprocessing, it kind of works:

$ cat groff.1 | perl -p -e 's/\\c\n//' > groff.test.1
$ LC_ALL=C po4a-updatepo -f man --no-deprecation --option groff_code=verbatim --option generated --option untranslated="}1,Ds,zY,zZ,Ee,ES,dT,FN,NE,NS,EX,EE,Id,rstReportMargin,INDENT,UNINDENT,UN,a.RE,\|" --option unknown_macros=untranslated --master groff.test.1 -M utf-8 -p test.pot
$ wc -l test.pot
3157 test.pot

I hope this helps you working on this, together with the discussion in
this bug.

Thanks for your support!

Greetings

Helge

[Helge Kreutzmann]

Hello Martin,

tag 1036826 fixed-upstream
thanks
Hello Helge,
I think I fixed this bug upstream, and it will be part of the next release,
later this month. I did not implement a full support for \c since it's
difficult in the current code base, but at least the groff.1 page proceeds.

This is very good news. Thank you very much!

If you have other failures from other pages, please tell me so that I can check
whether my fix is enough even before the release.

I can do so tomorrow afternoon, if this has time until then (I have
limited access atm).

Thanks for your help and patience,

The thanks is on my side!

Greetings

Helge

[Helge Kreutzmann]

Hello Martin,

tag 1036826 fixed-upstream
thanks
Hello Helge,
I think I fixed this bug upstream, and it will be part of the next release,
later this month. I did not implement a full support for \c since it's
difficult in the current code base, but at least the groff.1 page proceeds.

Thank you very much, I'm looking forward to the release.

If you have other failures from other pages, please tell me so that I can check
whether my fix is enough even before the release.

I checked with the documentation base from manpages-l10n. The
following pages are failing (with the current po4a):

* groff (as reported)
* credentials.7 (as of Debian unstable)
* mkosi.1 (only from OpenSUSE tumbleweed, page attached (compressed))
* afmtodit.1 (as of Debian unstable)

* And it also fail(ed) in mkpasswd.1 (as of Debian unstable), see
Debian #1036908

Thanks for your help and patience,

Again, thanks from my side.

Greetings

Helge

[Helge Kreutzmann]

Hello Martin,

Post by Helge Kreutzmann

tag 1036826 fixed-upstream
thanks
Hello Helge,
I think I fixed this bug upstream, and it will be part of the next release,
later this month. I did not implement a full support for \c since it's
difficult in the current code base, but at least the groff.1 page proceeds.

Thank you very much, I'm looking forward to the release.

If you have other failures from other pages, please tell me so that I can check
whether my fix is enough even before the release.

I checked with the documentation base from manpages-l10n. The
* groff (as reported)
* credentials.7 (as of Debian unstable)
* mkosi.1 (only from OpenSUSE tumbleweed, page attached (compressed))
* afmtodit.1 (as of Debian unstable)
* And it also fail(ed) in mkpasswd.1 (as of Debian unstable), see
Debian #1036908

Groff.1 works now, but all others listed here still fail. Is this
expected?

(Po4a 0.72 just hit unstable, so I tried them out).

Greetings

Helge

[Martin Quinson]

reopen 1036826
thanks

Hello,

Post by Helge Kreutzmann

Post by Helge Kreutzmann
I checked with the documentation base from manpages-l10n. The
* groff (as reported)
* credentials.7 (as of Debian unstable)
* mkosi.1 (only from OpenSUSE tumbleweed, page attached (compressed))
* afmtodit.1 (as of Debian unstable)
* And it also fail(ed) in mkpasswd.1 (as of Debian unstable), see
  Debian #1036908

Groff.1 works now, but all others listed here still fail. Is this
expected?

No it's not. I'll look into it.

Sorry,
Mt

[Helge Kreutzmann]

Hello Martin,

Post by Martin Quinson

Post by Helge Kreutzmann

Post by Helge Kreutzmann
I checked with the documentation base from manpages-l10n. The
* groff (as reported)
* credentials.7 (as of Debian unstable)
* mkosi.1 (only from OpenSUSE tumbleweed, page attached (compressed))
* afmtodit.1 (as of Debian unstable)
* And it also fail(ed) in mkpasswd.1 (as of Debian unstable), see
  Debian #1036908

Groff.1 works now, but all others listed here still fail. Is this
expected?

No it's not. I'll look into it.

Thank you very much!

Greetings

Helge

[Helge Kreutzmann]

Hello Martin,
gawk(1) is also affected by this bug (only the Debian version,
though).

Greetings

Helge

[Helge Kreutzmann]

Hello Martin,

Post by Helge Kreutzmann
Hello Martin,

If you have other failures from other pages, please tell me so that I can check
whether my fix is enough even before the release.

I can do so tomorrow afternoon, if this has time until then (I have
limited access atm).

No rush. If some problems presist, I'll try to fix it afterward.

Did you receive my e-mail from this morning, where I compiled the
remaining examples I'm aware of?

Greetings

Helge