Bug#1091995: tech-ctte: clarify authority of aliasing symbolic links

[Helmut Grohne]

Package: tech-ctte

Hello committee members,

I fear I am faced with another road block in the /usr-move transition
and seek assistance with a packaging authority question #1079329. The
fundamental question is who has authority to touch the aliasing symbolic
links such as /lib64. Until bookworm, there was no clear authority as
multiple packages (including but not limited to base-files, debootstrap,
glibc, systemd, usrmerge) touched them. As part of the /usr-move
transition, I moved authority of these links to base-files exclusively
as it now installs them via its data.tar. At least I thought so. systemd
maintainer Luca Boccassi disagrees and sees systemd having authority in
managing them. It has been demonstrated that this may cause an
installation failure of base-files in practical ways. In particular,
systemd assigns a different link target than base-files does. The
base-files.preinst has been augmented to catch this situation and abort
early to avoid a subsequent unpack error. Luca Boccassi argues that
base-files.preinst should fix up the link left behind by systemd. I
agree with that on the condition that we first fix the cause for the
wrong link (systemd), but Luca disagrees with changing systemd in this
regard. Doing so would deviate from upstream behaviour. As systemd
manages non-Debian containers, the management of /lib64 is necessary and
removing the link may break such container workloads (though a systemd
upstream MR to validate this claim turned up no actual breakage in
systemd's CI pipeline). As a result of this impasse, systemd continues
to set up a /lib64 that is incompatible with base-files and
base-files.preinst continues to fail installation.

I ask the technical committee to overrule the systemd maintainers and
authorizes me to NMU the proposed change[1] stopping systemd from
setting up an incompatible /lib64 symlink.

Let me stress that allowing /lib64 to point to different unpredictable
targets violates fundamental assumptions of the /usr-merge whose stated
goal was to treat /lib64 and /usr/lib64 interchangeably. Revoking this
assumption revokes the basis for the /usr-move transition that the CTTE
authorized me to carry out in repealing the moratorium.

Technically speaking, there is one more feasible workaround. If the
base-files package were to install an empty /usr/lib64 directory and the
/lib64 -> /usr/lib64 symbolic link in its data.tar for all
architectures, then the logic embodied in systemd now would never
trigger the offending code path for Debian installations and the problem
would likewise go away. I argue that this change has a significant risk
of confusing users of non-amd64 systems and therefore rejected it. You
may also overrule me and request that either base-files allows the link
to be unpredictable or unconditionally ships it for all architectures.

Both ways require a 3:1 majority vote via constitution 6.1.4.

Helmut

[1] https://github.com/systemd/systemd/pull/34804

[Simon McVittie]

Post by Helmut Grohne
Hello committee members,

(I am no longer a TC member, but I'm still on the mailing list)

Post by Helmut Grohne
In particular,
systemd assigns a different link target [for /lib64] than base-files does

If I was a TC member, the first question I would be asking is: what target?

I believe base-files creates /lib64 -> usr/lib64. Correct?

What conflicting symlink does systemd create under at least some
circumstances? Is it /lib64 -> usr/lib?

What architectures are affected by this? My reading of #1079329
is that it potentially affects any architecture that *does not*
have its canonical/interoperable ld.so(8) path in /lib64, so in
particular arm64 is usually affected (because arm64's canonical ld.so
is /lib/ld-linux-aarch64.so.1, so there is no reason why a minimal
arm64 system necessarily needs to contain /usr/lib64) but amd64 is not
(because amd64's canonical ld.so is /lib64/ld-linux-x86-64.so.2,
so every working usr-merged amd64 system must already contain
/usr/lib64/ld-linux-x86-64.so.2).

If https://wiki.debian.org/ArchitectureSpecificsMemo is accurate, the
loong64, mips64el, ppc64* and sparc64 architectures are in the same
equivalence class as amd64 (/lib64 exists as ABI), and all other known
official and -ports architectures are in the same equivalence class
as arm64.

smcv

[Helmut Grohne]

Hi Simon,

Post by Simon McVittie
(I am no longer a TC member, but I'm still on the mailing list)

I appreciate your feedback and asking questions that move discussions
forward.

Post by Simon McVittie
If I was a TC member, the first question I would be asking is: what target?
I believe base-files creates /lib64 -> usr/lib64. Correct?
What conflicting symlink does systemd create under at least some
circumstances? Is it /lib64 -> usr/lib?

I confirm all of the above.

Post by Simon McVittie
What architectures are affected by this? My reading of #1079329
is that it potentially affects any architecture that *does not*
have its canonical/interoperable ld.so(8) path in /lib64, so in
particular arm64 is usually affected (because arm64's canonical ld.so
is /lib/ld-linux-aarch64.so.1, so there is no reason why a minimal
arm64 system necessarily needs to contain /usr/lib64) but amd64 is not
(because amd64's canonical ld.so is /lib64/ld-linux-x86-64.so.2,
so every working usr-merged amd64 system must already contain
/usr/lib64/ld-linux-x86-64.so.2).
If https://wiki.debian.org/ArchitectureSpecificsMemo is accurate, the
loong64, mips64el, ppc64* and sparc64 architectures are in the same
equivalence class as amd64 (/lib64 exists as ABI), and all other known
official and -ports architectures are in the same equivalence class
as arm64.

I confirm your understanding. The equivalence class of amd64 is entirely
unaffected by this, because it always contains /usr/lib64 and in that
case, the link that systemd creates is compatible with base-files.

To see how this setup may pose practical problems, consider using an
ARM64 laptop. This system usually lacks /usr/lib64 and systemd creates
/lib64 -> /usr/lib. Now you install box64 or qemu-user and libc6:amd64
to run some amd64 application. This will not update /lib64 as it already
exists. Executing the binary now fails, because
/lib64/ld-linux-x86-64.so.2 does not exist. We installed it to
/usr/lib64 trusting that the symlink would make it visible in the
required location.

This very much is a Debian-specific problem, because Debian is the root
of the only Linux distribution hierarchy that does multiarch and allows
pretty much arbitrary combinations of architectures. Everywhere else
you'd rather set up a container where there would be a separate /lib64
that systemd would setup in a different way. The systemd assumption here
is that /lib64 can be architecture-dependent, but that assumption is
rendered invalid by multiarch.

I think there is one more way out here. We may choose to install
/usr/lib64 as a symlink to lib (e.g. in base-files' data.tar) and then
move ld-linux-x86-64.so.2 to /usr/lib in libc6:amd64. At that point, we
no longer care whether /lib64 points to /usr/lib64 or /usr/lib (both
work equally well). This actually is rather close to what Arch Linux
does. Going this route requires killing multilib as the multilib
libraries (e.g. libc6-amd64:i386) would then collide with system
libraries. I don't think I'd miss multilib as our cross toolchains have
matured significantly, but I expect others to disagree and you're back
to overriding a different developer.

As far as I understand it, we may pick any three:
* Allow systemd to manage /lib64 as it does.
* /usr-move
* Multiarch
* Multilib

No matter how you choose here, one of these will be broken.

Helmut

[Helmut Grohne]

Hi Josh,

Post by Helmut Grohne
* Allow systemd to manage /lib64 as it does.
* /usr-move
* Multiarch
* Multilib

By "/usr-move", do you mean the overall /usr migration, or do you mean
the idea of linking /lib64 to /usr/lib64 unconditionally (e.g. in
base-files)? The latter seems like a viable solution, and a *useful* one
for multiarch. If someone is on arm64 (without /lib64), and they want to
have amd64 libraries installed, *something* needs to be responsible for
installing a /lib64 symlink to /usr/lib64, and base-files seems like the
obvious candidate to do so. That would avoid having to have any special
logic in packages that need /lib64. The only cost would be having an
"unnecessary" symlink on targets like arm64 (which stops being
unnecessary as soon as the system installs multiarch packages from a
target that needs /lib64).

I would say that the answer is more nuanced.

When I say /usr-move, what I mean is emptying directories such as /bin,
/lib and /sbin but also /lib64 and moving all of their files to the
corresponding /usr location as a mechanism to avoid aliasing effects
from surfacing in dpkg. As such, /usr-move is an extension of
/usr-merge, which merely says that /bin, /lib, /sbin and a few more are
symbolic links. (Let's forget about the alternative merge via symlink
farms for now.) The implication of /usr-move is that libc6:amd64
installs /usr/lib64/ld-linux-x86-64.so.2 rather than
/lib64/ld-linux-x86-64.so.2 and expects /lib64 to always exist.

Your suggestion of having these links managed by base-files is already
implemented. In trixie and later, base-files installs /bin, /lib, /sbin
as symlinks via its data.tar. On amd64, it also installs /lib64 that
way, but not on arm64. (Please consider these equivalence classes with
an eye towards Simon's mail.) If you install libc6:amd64 on an arm64
installation, it will unpack /usr/lib64. This unpack happens to activate
a base-files trigger and base-files will subsequently create the /lib64
symlink. I think we already are where you want us to be as you may
practically assume /lib64 to exist whenever any package installs
anything into /usr/lib64.

Unconditionally installing an empty /usr/lib64 (and /lib64) for all
architectures is something that would technically work indeed. Once
/usr/lib64 exists everywhere, the logic in systemd will never create a
/lib64 -> usr/lib symbolic link and the dispute is resolved. I happen to
disagree with this being a good solution, but the CTTE may overrule me
here.

Helmut

[Matthew Vernon]

Hi,

Luca, I hope you don't mind me CC'ing you on this mail, but I'd like to
check I've understood your objections to Helmut's plan correctly.

Helmut wants systemd to stop making a /lib64 -> /usr/lib link on arm64.
As I understand it, your objections to that are:

1) Removing this link may break some container workflows
2) Systemd upstream rejected the change
3) dracut (where Helmut found this issue) is not an official bootstrap
tool, and the problem doesn't occur when using initramfs-tools

Is that correct?

Regards,

Matthew

[Matthew Garrett]

I call for votes on the below ballot. The vote is open for 7 days, or
until the outcome is beyond doubt.

In Bug #1091995, the Technical Committe was asked to rule on an issue
that could, under certain circumstances, result in failure of the
base-files package to install or upgrade correctly. Under these
circumstances, systemd will create a symlink from /lib64 to /usr/lib,
which does not match the symlink contained within base-files. base-files
will detect this case in preinst and generate an error, but if it did
not do this then dpkg would instead fail with a less verbose message.

Policy does not currently define ownership of the usrmerge filesystem
aliases, but since trixie base-files has effectively been responsible
for ensuring that these aliases are configured appropriately. This is
therefore a technical disagreement rather than a policy violation.

A) The Technical Committee affirms that base-files should own all
top-level filesystem aliases, and packages that conflict with this must
be patched in Debian to avoid creating any aliases that conflict with
base-files (overrules the systemd maintainer, requires 3:1 majority
vote)

B) The Technical Committee requests that base-files create an empty
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,
requires 3:1 majority vote)

C) The Technical Committee requests that base-files preinst check
whether /lib64 is a symlink to /usr/lib and, if so, replace it with a
symlink to /usr/lib64 (overrules the base-files maintainer, requires 3:1
majority vote)

N) None of the above / Further Discussion

[Helmut Grohne]

Post by Matthew Garrett
In Bug #1091995, the Technical Committe was asked to rule on an issue
that could, under certain circumstances, result in failure of the
base-files package to install or upgrade correctly. Under these
circumstances, systemd will create a symlink from /lib64 to /usr/lib,
which does not match the symlink contained within base-files. base-files
will detect this case in preinst and generate an error, but if it did
not do this then dpkg would instead fail with a less verbose message.

Given that I have raised this issue to the committee, I have a conflict
of interest on this matter. As a result, I explicitly abstain from
voting on this matter.

This may be relevant in case, five out of seven members have voted in
favour of one option. Given my explicit non-vote, the vote may no longer
be in doubt at that time.

Helmut

[Christoph Berg]

Re: Matthew Garrett

Post by Matthew Garrett
B) The Technical Committee requests that base-files create an empty

^^^^^^^^^^

Post by Matthew Garrett
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,

^^^^^^^^^^

I guess that should be "systemd"?

Christoph

[Matthew Vernon]

Post by Christoph Berg
Re: Matthew Garrett

Post by Matthew Garrett
B) The Technical Committee requests that base-files create an empty

^^^^^^^^^^

Post by Matthew Garrett
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,

^^^^^^^^^^
I guess that should be "systemd"?

No; the effect of this change would be that /usr/lib64 would always
exist, so when systemd makes the /lib64 symlink, it will make it to
/usr/lib64 rather than /usr/lib; that creation of /lib64 -> /usr/lib is
the problematic behaviour of systemd at the moment. Making this change
would mean that systemd's behaviour (/lib64 -> /usr/lib64) is what
base-files does at the moment.

Regards,

Matthew

[Simon McVittie]

Post by Christoph Berg
Re: Matthew Garrett

Post by Matthew Garrett
B) The Technical Committee requests that base-files create an empty

^^^^^^^^^^

Post by Matthew Garrett
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,

^^^^^^^^^^
I guess that should be "systemd"?

No; the effect of this change would be that /usr/lib64 would always exist,
so when systemd makes the /lib64 symlink, it will make it to /usr/lib64
rather than /usr/lib; that creation of /lib64 -> /usr/lib is the problematic
behaviour of systemd at the moment.

Specifically, systemd is already working fine on architectures where
/usr/lib64 is required to exist anyway (like amd64 and riscv64), and
the problematic behaviour only occurs on architectures where /usr/lib64
doesn't exist (like for example armel and i386). Option B would be a
way for base-files to nudge systemd into having the same behaviour on
the second category of architectures that it currently does on the first
category of architectures, without systemd code changes.

smcv

[Christoph Berg]

Re: Matthew Vernon

Post by Christoph Berg
Re: Matthew Garrett

Post by Matthew Garrett
B) The Technical Committee requests that base-files create an empty

^^^^^^^^^^

Post by Matthew Garrett
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,

^^^^^^^^^^
I guess that should be "systemd"?

No

Oh sorry, I overlooked the "If systemd creates..." part. Now it makes
sense.

Christoph

[Christoph Berg]

Re: Matthew Garrett

Post by Matthew Garrett
A) The Technical Committee affirms that base-files should own all
top-level filesystem aliases, and packages that conflict with this must
be patched in Debian to avoid creating any aliases that conflict with
base-files (overrules the systemd maintainer, requires 3:1 majority
vote)
B) The Technical Committee requests that base-files create an empty
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,
requires 3:1 majority vote)
C) The Technical Committee requests that base-files preinst check
whether /lib64 is a symlink to /usr/lib and, if so, replace it with a
symlink to /usr/lib64 (overrules the base-files maintainer, requires 3:1
majority vote)
N) None of the above / Further Discussion

I vote

A > N > B = C

Christoph

[Stefano Rivera]

Hi Matthew (2025.02.25_16:59:44_-0400)

Post by Matthew Garrett
I call for votes on the below ballot. The vote is open for 7 days, or
until the outcome is beyond doubt.
In Bug #1091995, the Technical Committe was asked to rule on an issue
that could, under certain circumstances, result in failure of the
base-files package to install or upgrade correctly. Under these
circumstances, systemd will create a symlink from /lib64 to /usr/lib,
which does not match the symlink contained within base-files. base-files
will detect this case in preinst and generate an error, but if it did
not do this then dpkg would instead fail with a less verbose message.
Policy does not currently define ownership of the usrmerge filesystem
aliases, but since trixie base-files has effectively been responsible
for ensuring that these aliases are configured appropriately. This is
therefore a technical disagreement rather than a policy violation.
A) The Technical Committee affirms that base-files should own all
top-level filesystem aliases, and packages that conflict with this must
be patched in Debian to avoid creating any aliases that conflict with
base-files (overrules the systemd maintainer, requires 3:1 majority
vote)
B) The Technical Committee requests that base-files create an empty
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,
requires 3:1 majority vote)
C) The Technical Committee requests that base-files preinst check
whether /lib64 is a symlink to /usr/lib and, if so, replace it with a
symlink to /usr/lib64 (overrules the base-files maintainer, requires 3:1
majority vote)
N) None of the above / Further Discussion

I vote:

A > N > B = C

Rationale:

This really seems like a systemd bug in violation of the way we
usr-merge. I could see it making sense on distributions without
multiarch, but not Debian. We could obviously live with this and
work-around it in other packages, if necessary. But I haven't seen any
compelling reason for needing to do that.

Ideally this would be resolved upstream in systemd, as this affects
Debian containers in systemd-nspawn on other distributions.

I vote B and C equally, as I believe the base-files maintainers can
select between these options (that they disagree with) themselves.

Stefano

[Timo Röhling]

Hi,

Post by Matthew Garrett
A) The Technical Committee affirms that base-files should own all
top-level filesystem aliases, and packages that conflict with this must
be patched in Debian to avoid creating any aliases that conflict with
base-files (overrules the systemd maintainer, requires 3:1 majority
vote)
B) The Technical Committee requests that base-files create an empty
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,
requires 3:1 majority vote)
C) The Technical Committee requests that base-files preinst check
whether /lib64 is a symlink to /usr/lib and, if so, replace it with a
symlink to /usr/lib64 (overrules the base-files maintainer, requires 3:1
majority vote)
N) None of the above / Further Discussion

I vote

A > N > B = C


Cheers
Timo

[Matthew Garrett]

Post by Matthew Garrett
A) The Technical Committee affirms that base-files should own all
top-level filesystem aliases, and packages that conflict with this must
be patched in Debian to avoid creating any aliases that conflict with
base-files (overrules the systemd maintainer, requires 3:1 majority
vote)
B) The Technical Committee requests that base-files create an empty
/usr/lib64 directory, even on architectures that do not use lib64. If
systemd creates a symlink, this will then match the behaviour of
base-files and avoid the issue (overrules the base-files maintainer,
requires 3:1 majority vote)
C) The Technical Committee requests that base-files preinst check
whether /lib64 is a symlink to /usr/lib and, if so, replace it with a
symlink to /usr/lib64 (overrules the base-files maintainer, requires 3:1
majority vote)
N) None of the above / Further Discussion

I vote A > N > B = C

[Craig Small]

A > B > C > N

Base-files is the right place for this directory but if there needs to be
some sort of fix, then I prefer B over C.

- Craig

[Matthew Vernon]

Hi,

The TC vote ranks A unanimously first (excepting Helmut, who abstained
due to a conflict of interest). The resolution is, therefore, as follows:

In Bug #1091995, the Technical Committe was asked to rule on an issue
that could, under certain circumstances, result in failure of the
base-files package to install or upgrade correctly. Under these
circumstances, systemd will create a symlink from /lib64 to /usr/lib,
which does not match the symlink contained within base-files. base-files
will detect this case in preinst and generate an error, but if it did
not do this then dpkg would instead fail with a less verbose message.

Policy does not currently define ownership of the usrmerge filesystem
aliases, but since trixie base-files has effectively been responsible
for ensuring that these aliases are configured appropriately. This is
therefore a technical disagreement rather than a policy violation.

The Technical Committee affirms that base-files should own all
top-level filesystem aliases, and packages that conflict with this must
be patched in Debian to avoid creating any aliases that conflict with
base-files. In doing so, the Technical Committee overrides the systemd
maintainers.

Regards,

Matthew