Bug#1079329: systemd(?) still creates /lib64 on arm64

[Luca Boccassi]

Control: tags -1 - moreinfo
What further information do you require exactly?

That's correct, as the dynamic loader is at /usr/lib/ld-linux-
aarch64.so.1 and the symlink will be created to the first directory
between /usr/lib64 and /usr/lib that exists and has the loader, in that
order.
So it seems to me either base-files needs to be able to deal with this,
or the loader should be symlinked in /usr/lib64 (like Fedora does)?

I see no technical reason for base-files to deal with packages messing
with /lib64 symlinks. The pending policy change forbids doing so and you
seconded it.

As far as I can tell, unless I looked up the wrong thing, the policy
change specifically refers to packages installing files, which is not
what is happening here: the packages are not installing anything,
neither directly nor via maintainer scripts, it's purely a runtime
functionality. And I think it would be wrong if it applied to this
too: this runtime functionality is needed, and you have explained the
use case yourself below, so I do not believe policy should forbid it.

I also see no reason to add the loader to /usr/lib64 as nothing
references that path nor any reason for having /lib64 on arm64 in the
first place.
Please clarify what kind of problem is being solved on the systemd side
in attempting to manage a /lib64 symlink. I can partially answer this
question myself: systemd supports "hermetic /usr" which amounts to
bootstrapping an installation from a bare /usr tree. As such, it
contains code to manage locations outside /usr and that happens to
include aliasing symlinks. What I do not see is why it has to manage
/lib64 on arm64. It could simply stop doing so and all would be good as
far as I can see. Let us pretend systemd upstream were to drop the code
for creating /lib64 on arm64. Which users would complain?
Helmut

Yes that's the use case, just as you defined it. The problem is that
these links need to exist, and they need to be created at runtime in
an ephemeral root, and the target needs to be figured out <somehow>.
The best we have so far, that has been working fine over the years, is
to target it to where the loader is located. If you think there's a
better way, that works everywhere, please feel free to propose such a
change upstream, via a pull request ideally, or at least via an issue
(but please do note that PRs are better). In the end I think that's
the right place to discuss such details, rather than a downstream bug.

[Helmut Grohne]

Hi Luca,

Post by Luca Boccassi
Yes that's the use case, just as you defined it. The problem is that
these links need to exist, and they need to be created at runtime in
an ephemeral root, and the target needs to be figured out <somehow>.
The best we have so far, that has been working fine over the years, is
to target it to where the loader is located. If you think there's a
better way, that works everywhere, please feel free to propose such a
change upstream, via a pull request ideally, or at least via an issue
(but please do note that PRs are better). In the end I think that's
the right place to discuss such details, rather than a downstream bug.

Thanks for confirming. Please go ahead and remove the creation of /lib64
from systemd then.

Helmut

[Luca Boccassi]

Post by Helmut Grohne
Hi Luca,

Post by Luca Boccassi
Yes that's the use case, just as you defined it. The problem is that
these links need to exist, and they need to be created at runtime in
an ephemeral root, and the target needs to be figured out <somehow>.
The best we have so far, that has been working fine over the years, is
to target it to where the loader is located. If you think there's a
better way, that works everywhere, please feel free to propose such a
change upstream, via a pull request ideally, or at least via an issue
(but please do note that PRs are better). In the end I think that's
the right place to discuss such details, rather than a downstream bug.

Thanks for confirming. Please go ahead and remove the creation of /lib64
from systemd then.
Helmut

Sorry, I do not follow. As mentioned, this cannot just be removed.

[Luca Boccassi]

Post by Helmut Grohne
Hi Luca,

Post by Luca Boccassi

Post by Helmut Grohne

Post by Luca Boccassi
Yes that's the use case, just as you defined it. The problem is that
these links need to exist, and they need to be created at runtime in
an ephemeral root, and the target needs to be figured out <somehow>.
The best we have so far, that has been working fine over the years, is
to target it to where the loader is located. If you think there's a
better way, that works everywhere, please feel free to propose such a
change upstream, via a pull request ideally, or at least via an issue
(but please do note that PRs are better). In the end I think that's
the right place to discuss such details, rather than a downstream bug.

Thanks for confirming. Please go ahead and remove the creation of /lib64
from systemd then.

Sorry, I do not follow. As mentioned, this cannot just be removed.

You appeared to agree with my general description of the mechanism. You
detailed that systemd needs to create these links when needed at
runtime, but for arm64 there is no known nor documented need for /lib64,
so I read that as you agreeing the /lib64 link on arm64 should be
removed upstream. If you believe that it cannot be removed, please give
details why you think so.

Sorry, but I am still not following. I do not understand, why would
lib64 not be needed? It's a 64bit architecture, no? If I download a
Fedora 40 aarch64 image from:

https://fedoraproject.org/server/download

and check the root directory, this is what I see:

total 16
dr-xr-xr-x. 2 root root 6 Jan 24 2024 afs
lrwxrwxrwx. 1 root root 7 Jan 24 2024 bin -> usr/bin
drwxr-xr-x. 2 root root 6 Apr 15 00:02 boot
drwxr-xr-x. 2 root root 6 Apr 15 00:02 dev
drwxr-xr-x. 121 root root 8192 Apr 15 00:26 etc
drwxr-xr-x. 2 root root 6 Jan 24 2024 home
lrwxrwxrwx. 1 root root 7 Jan 24 2024 lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Jan 24 2024 lib64 -> usr/lib64
drwxr-xr-x. 2 root root 6 Jan 24 2024 media
drwxr-xr-x. 2 root root 6 Jan 24 2024 mnt
drwxr-xr-x. 2 root root 6 Jan 24 2024 opt
drwxr-xr-x. 2 root root 6 Apr 15 00:02 proc
dr-xr-x---. 3 root root 126 Apr 15 00:27 root
drwxr-xr-x. 2 root root 6 Apr 15 00:02 run
lrwxrwxrwx. 1 root root 8 Jan 24 2024 sbin -> usr/sbin
drwxr-xr-x. 2 root root 6 Jan 24 2024 srv
drwxr-xr-x. 2 root root 6 Apr 15 00:02 sys
drwxrwxrwt. 2 root root 6 Apr 15 00:02 tmp
drwxr-xr-x. 12 root root 144 Apr 15 00:03 usr
drwxr-xr-x. 20 root root 4096 Apr 15 00:12 var

Post by Helmut Grohne
If you insist on cargo culting this code, there is another way. We know
for a fact that in the hermetic-/usr case, there is no package manager
on whose toes systemd could step. In particular, there is no
/var/lib/dpkg. So systemd could simply check that location and skip
bothering with aliasing links if it exists. Thus we would turn the
broken code in systemd into dead code on Debian systems until you
transform the installation hermetic and then it would return.

Thanks for the suggestion. This might not be necessarily the case
though, there can still be (and in fact there likely will) be a /var/
partition, even if / is a tmpfs. More importantly, I'm afraid this is
sort of debianism has no chance of being accepted upstream, sorry.

[Helmut Grohne]

Hi Luca,

But how can you be sure that nothing depends on it? How have you
checked that it's not needed? This logic has been there for many years

You can never be sure. If that were your bar for removing compatibility
(and I know it is not), you could never remove any compatibility. You'd
paint yourself in a corner where you can longer modify the code base for
the fear of breaking something. Evidently, that is not how systemd is
being maintained. Instead, it breaks (and fixes) stuff frequently and
honestly that's the only way of making progress.

I have strong clues however. A default bookworm installation on arm64
does not have /lib64. I also fed the necessary change into systemd's
upstream CI https://github.com/systemd/systemd/pull/34804. It produced
five failures all of which are on amd64 and unrelated. In particular,
the ppc64el, s390x and arm64 ones that are affected, are ok.

On the flip side, you repeatedly side stepped the reverse question: What
is the purpose of these links? I am still missing an answer here.

The way this would be approached on the Linux kernel side is merging the
change given sufficient evidence (and I think the above CI results
combined with the persistent lack of an answer from your side poses such
evidence) and seeing what breaks. When stuff breaks, the change would be
reverted recording what kind of use case needs the code (thus answering
my question). Do you see any reason for not using the same approach for
such a change in systemd?

The fundamental disagreement seems to be this:
* You don't want to remove these links on the systemd side unless there
is a proof for them to never be needed. (<- cannot be proven)
* I don't want to work around them on the base-files side unless there
is evidence for them being needed. (<- no evidence presented thus
far)

Do you see any other way of resolving this? This is a classical overlap
of responsibility problem. If all else fails, we might ask the CTTE if
you agree.

Helmut