Bug#1094583: libvirt-daemon-driver-qemu: apparmor template missing from filesystem

[Alban Browaeys]

Package: libvirt-daemon-driver-qemu
Version: 11.0.0-1
Severity: grave
Justification: renders package unusable
Dear Maintainer,
/etc/apparmor.d/libvirt/TEMPLATE.qemu
total 0

I cannot reproduce
ii libvirt-daemon-driver-qemu 11.0.0-1 amd64 Virtualization daemon QEMU connection driver

ls -l /etc/apparmor.d/libvirt/TEMPLATE.qemu
-rw-r--r-- 1 root root 192 2 sept. 11:47 /etc/apparmor.d/libvirt/TEMPLATE.qemu

Either way if the template is shipped by the package it is not a package bug if the file is missing after installation.
Still it could be an dpkg/apt bug but unlikely.

Could it be you were running out of space on the /etc partition while installing or had a crash that corrupted
this filesystem while installing ?

https://packages.debian.org/trixie/amd64/libvirt-daemon-driver-qemu/filelist shows the apparmor template is shipped

downloading
http://http.us.debian.org/debian/pool/main/libv/libvirt/libvirt-daemon-driver-qemu_11.0.0-1_amd64.deb
and opening it with file-roller shows inside of it an /etc/apparmor.d/libvirt/TEMPLATE.qemu file with content: "
#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
#include <abstractions/libvirt-qemu>
}
"

This bug looks like a local system issue.

Cheers,
Alban

Unable to complete install: 'internal error: cannot load AppArmor profile 'libvirt-f9987331-aa46-412e-baf0-bdef4b5a631e''
   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 71, in cb_wrapper
     callback(asyncjob, *args, **kwargs)
     ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/share/virt-manager/virtManager/createvm.py", line 2008, in _do_async_install
     installer.start_install(guest, meter=meter)
     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
   File "/usr/share/virt-manager/virtinst/install/installer.py", line 726, in start_install
     domain = self._create_guest(
             guest, meter, initial_xml, final_xml,
             doboot, transient)
   File "/usr/share/virt-manager/virtinst/install/installer.py", line 667, in _create_guest
     domain = self.conn.createXML(initial_xml or final_xml, 0)
   File "/usr/lib/python3/dist-packages/libvirt.py", line 4545, in createXML
     raise libvirtError('virDomainCreateXML() failed')
libvirt.libvirtError: internal error: cannot load AppArmor profile 'libvirt-f9987331-aa46-412e-baf0-bdef4b5a631e'
2025-01-28T11:21:15.809798-05:00 saratoga libvirtd[1025]: internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/virt-aa-helper -c -u lib
virt-f9987331-aa46-412e-baf0-bdef4b5a631e) unexpected exit status 1: virt-aa-helper: error: template does not exist#012virt-aa-helper: error: could not create
profile
2025-01-28T11:21:15.809885-05:00 saratoga libvirtd[1025]: internal error: cannot load AppArmor profile 'libvirt-f9987331-aa46-412e-baf0-bdef4b5a631e'
Debian Release: trixie/sid
   APT prefers testing
   APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 6.12.10-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
ii  adduser                     3.137
ii  debconf [debconf-2.0]       1.5.89
ii  libc6                       2.40-6
ii  libgcc-s1                   14.2.0-12
ii  libglib2.0-0t64             2.82.4-2

[Andrea Bolognani]

The package manifest includes an AppArmor template, but it is
# dpkg -L libvirt-daemon-driver-qemu | grep -i template
/etc/apparmor.d/libvirt/TEMPLATE.qemu
# ls -l /etc/apparmor.d/libvirt/
total 0

Hey Kevin,

/etc/apparmor.d/libvirt/TEMPLATE.qemu [file not found]
/etc/libvirt/qemu-lockd.conf [file not found]
/etc/libvirt/qemu-sanlock.conf [file not found]
/etc/libvirt/qemu.conf [Errno 13] Permission denied: '/etc/libvirt/qemu.conf'
/etc/logrotate.d/libvirtd.qemu [file not found]

That's very interesting because that almost completely overlaps with
the list of conffiles that have been moved from the
libvirt-daemon-system package, where they historically lived, to the
libvirt-daemon-driver-qemu package.

We have special handling in maintainer scripts for those, and though
it should be fairly battle-tested by now it's not impossible that we
would have some lingering bugs.

However...

This was an upgrade from the previous version in testing, so it may be
something to be aware of in the upgrade process.

... this is surprising, since the files were moved in 10.6.0-2. If
you were upgrading from the previous version in testing (10.10.0-3),
that special handling shouldn't have been triggered.

I'll ask you to please verify a few things for me. You should be able
to gather all the information by looking at /var/log/apt/term.log*.

What version of libvirt did you have installed before the upgrade?

Do you have the libvirt-daemon-system package installed? And was it
installed before the upgrade?

Was any message containing the string "transfer of config file" shown
during the upgrade?

What does the output of

$ sudo find /etc/libvirt/ -name '*dpkg*'

look like?

Thanks in advance!

[Andrea Bolognani]

Thanks for the pointer to /var/log/apt/term.log*
On a previous upgrade to 10.7.0-3 from 9.0.0-4+deb12u1, I see a bunch of
messages about "Preparing transfer of config file ... (from
libvirt-daemon-system to X)" where X included libvirt-daemon-common,
libvirt-daemon-driver-{xen,lxx,qemu,network}.
It looks like that transfer failed somehow because there are a bunch of
{conffile}.dpkg-transfer files on the filesystem.

Okay, that tracks. It's surprising that you were able to start VMs at
all after that point though, I would have expected trouble to have
begun back then.

Can you please provide the full log for the relevant apt transaction?

$ sudo find /etc/libvirt/ -name '*dpkg*'
/etc/libvirt/qemu-lockd.conf.dpkg-transfer
/etc/libvirt/qemu.conf.dpkg-old
/etc/libvirt/libvirtd.conf.dpkg-transfer
/etc/libvirt/network.conf.dpkg-transfer
/etc/libvirt/qemu.conf.dpkg-transfer
/etc/libvirt/qemu-sanlock.conf.dpkg-transfer
/etc/libvirt/lxc.conf.dpkg-transfer
/etc/libvirt/libxl-sanlock.conf.dpkg-transfer
/etc/libvirt/libxl-lockd.conf.dpkg-transfer
/etc/libvirt/virtlogd.conf.dpkg-transfer
/etc/libvirt/virtlockd.conf.dpkg-transfer
/etc/libvirt/libxl.conf.dpkg-transfer

Thanks. Can you please provide the output of

$ sudo find /etc/libvirt/ -ls | grep -i conf
$ sudo find /etc/apparmor.d/ -ls | grep -i virt

so that I can get a clearer picture? I should have asked for this to
begin with, sorry :)

I wonder if the transfer was impacted by /etc/libvirt/qemu.conf having been
modified. At the prompt I chose "Y: install the package maintainer's
version" to ease the upgrade with the intent to go back re-modify it later.

It's weird that you got the prompt at all: if I'm reading the code
correctly, that shouldn't have happened. Of course that's arguably in
itself a problem...

[Kevin Otte]

Since this is a test machine, I don't think I'd tried to start a VM
until this week. This only came up because I finally had some time to
work on a different VM related bug.

I was going to try and strip out only the relevant transaction, but it
looks like there were multiple and I'm not entirely certain if they
relate, so I've attached the whole log file. There's probably a lot of
noise, but I'd rather provide too much info than omit something important.

I'm used to the changed file prompt on /etc/libvirt/qemu.conf as I
usually get it even on stable updates. I add a `spice_listen = "::1"` in
there to force IPv6 for my environment.

Unfortunately I did start to flail a bit to try and get things going
again, so the outputs may not be quite as useful. Not knowing what those
transfer files were and seeing fairly old timestamps on them, I deleted
the whole /etc/apparmor.d/libvirt directory in hopes of having an `apt
--reinstall` regenerate it. I then also ran a `dpkg --force-confmiss -i
/tmp/libvirt-daemon-driver-qemu_11.0.0-1_amd64.deb` to get the missing
files back. I did not do this for any of the other packages, so those
file states may still be of use:

***@saratoga:~$ sudo find /etc/libvirt/ -ls | grep -i conf
131543 4 -rw-r--r-- 1 root root 2169 Jul 1 2022
/etc/libvirt/qemu-lockd.conf.dpkg-transfer
131321 40 -rw------- 1 root root 39031 Dec 17 14:58
/etc/libvirt/qemu.conf.dpkg-old
132450 4 -rw-r--r-- 1 root root 1041 Jan 17 17:39
/etc/libvirt/network.conf
132849 4 -rw-r--r-- 1 root root 547 Sep 2 05:47
/etc/libvirt/libvirt.conf
132617 20 -rw-r--r-- 1 root root 17826 Feb 26 2023
/etc/libvirt/libvirtd.conf.dpkg-transfer
132860 4 -rw-r--r-- 1 root root 1041 Sep 16 15:41
/etc/libvirt/network.conf.dpkg-transfer
131527 4 -rw-r--r-- 1 root root 2169 Jan 17 17:39
/etc/libvirt/qemu-lockd.conf
132821 40 -rw------- 1 root root 37466 Sep 16 15:41
/etc/libvirt/qemu.conf.dpkg-transfer
131820 40 -rw------- 1 root root 39126 Jan 28 11:18
/etc/libvirt/qemu.conf
131544 4 -rw-r--r-- 1 root root 2465 Jul 1 2022
/etc/libvirt/qemu-sanlock.conf.dpkg-transfer
131541 4 -rw-r--r-- 1 root root 1175 Jul 1 2022
/etc/libvirt/lxc.conf.dpkg-transfer
131539 4 -rw-r--r-- 1 root root 2465 Jul 1 2022
/etc/libvirt/libxl-sanlock.conf.dpkg-transfer
131538 4 -rw-r--r-- 1 root root 2169 Jul 1 2022
/etc/libvirt/libxl-lockd.conf.dpkg-transfer
132847 4 -rw-r--r-- 1 root root 4095 Sep 2 05:47
/etc/libvirt/virtlogd.conf.dpkg-transfer
132848 4 -rw-r--r-- 1 root root 450 Sep 2 05:47
/etc/libvirt/libvirt-admin.conf
131528 4 -rw-r--r-- 1 root root 2465 Jan 17 17:39
/etc/libvirt/qemu-sanlock.conf
131546 4 -rw-r--r-- 1 root root 3058 Jul 1 2022
/etc/libvirt/virtlockd.conf.dpkg-transfer
131540 4 -rw-r--r-- 1 root root 2268 Jul 1 2022
/etc/libvirt/libxl.conf.dpkg-transfer
***@saratoga:~$ sudo find /etc/apparmor.d/ -ls | grep -i virt
131519 4 -rw-r--r-- 1 root root 1807 Sep 16 15:41
/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper.dpkg-transfer
131779 0 -rw-r--r-- 1 root root 0 Aug 31 2022
/etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper
131784 0 -rw-r--r-- 1 root root 0 Aug 31 2022
/etc/apparmor.d/local/usr.sbin.libvirtd
131999 4 -rw-r--r-- 1 root root 1898 Jan 17 17:39
/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper
132803 8 -rw-r--r-- 1 root root 4735 Sep 16 15:41
/etc/apparmor.d/usr.sbin.libvirtd.dpkg-transfer
132574 8 -rw-r--r-- 1 root root 4780 Jan 17 17:39
/etc/apparmor.d/usr.sbin.libvirtd
786438 4 drwxr-xr-x 2 root root 4096 Jan 30 16:43
/etc/apparmor.d/libvirt
786469 4 -rw-r--r-- 1 root root 303 Jan 28 21:18
/etc/apparmor.d/libvirt/libvirt-88e89a0c-232f-4abc-b3ac-56cd87b1a707
786471 4 -rw-r--r-- 1 root root 303 Jan 28 21:21
/etc/apparmor.d/libvirt/libvirt-328ff1d4-b514-4d66-be07-8403e4c206b0
786478 4 -rw-r--r-- 1 root root 303 Jan 28 21:35
/etc/apparmor.d/libvirt/libvirt-d736d34f-ec0c-476c-b320-b198ae86bb47
786482 4 -rw-r--r-- 1 root root 303 Jan 30 11:23
/etc/apparmor.d/libvirt/libvirt-ff762a15-9f28-491e-9bec-4ef668fa277f
786461 4 -rw-r--r-- 1 root root 192 Jan 15 03:06
/etc/apparmor.d/libvirt/TEMPLATE.qemu
132513 12 -rw-r--r-- 1 root root 9258 Jan 17 17:39
/etc/apparmor.d/abstractions/libvirt-qemu
132798 8 -rw-r--r-- 1 root root 4610 Sep 16 15:41
/etc/apparmor.d/abstractions/libvirt-lxc.dpkg-transfer
132809 12 -rw-r--r-- 1 root root 9063 Sep 16 15:41
/etc/apparmor.d/abstractions/libvirt-qemu.dpkg-transfer

Post by Andrea Bolognani

Thanks for the pointer to /var/log/apt/term.log*
On a previous upgrade to 10.7.0-3 from 9.0.0-4+deb12u1, I see a bunch of
messages about "Preparing transfer of config file ... (from
libvirt-daemon-system to X)" where X included libvirt-daemon-common,
libvirt-daemon-driver-{xen,lxx,qemu,network}.
It looks like that transfer failed somehow because there are a bunch of
{conffile}.dpkg-transfer files on the filesystem.

Okay, that tracks. It's surprising that you were able to start VMs at
all after that point though, I would have expected trouble to have
begun back then.
Can you please provide the full log for the relevant apt transaction?

$ sudo find /etc/libvirt/ -name '*dpkg*'
/etc/libvirt/qemu-lockd.conf.dpkg-transfer
/etc/libvirt/qemu.conf.dpkg-old
/etc/libvirt/libvirtd.conf.dpkg-transfer
/etc/libvirt/network.conf.dpkg-transfer
/etc/libvirt/qemu.conf.dpkg-transfer
/etc/libvirt/qemu-sanlock.conf.dpkg-transfer
/etc/libvirt/lxc.conf.dpkg-transfer
/etc/libvirt/libxl-sanlock.conf.dpkg-transfer
/etc/libvirt/libxl-lockd.conf.dpkg-transfer
/etc/libvirt/virtlogd.conf.dpkg-transfer
/etc/libvirt/virtlockd.conf.dpkg-transfer
/etc/libvirt/libxl.conf.dpkg-transfer

Thanks. Can you please provide the output of
$ sudo find /etc/libvirt/ -ls | grep -i conf
$ sudo find /etc/apparmor.d/ -ls | grep -i virt
so that I can get a clearer picture? I should have asked for this to
begin with, sorry :)

I wonder if the transfer was impacted by /etc/libvirt/qemu.conf having been
modified. At the prompt I chose "Y: install the package maintainer's
version" to ease the upgrade with the intent to go back re-modify it later.

It's weird that you got the prompt at all: if I'm reading the code
correctly, that shouldn't have happened. Of course that's arguably in
itself a problem...

[Andrea Bolognani]

Control: severity -1 important

Post by Andrea Bolognani
Okay, that tracks. It's surprising that you were able to start VMs at
all after that point though, I would have expected trouble to have
begun back then.

Sorry, working through the brain fog a bit: I was able to start an existing
VM, probably because the apparmor profile with its UUID still existed. I
noticed the issue because I deleted the VM to try and recreate it, which
failed because the template didn't exist.

This makes sense. I'll have a better look at the additional
information you've provided tomorrow and try to figure out what went
wrong during the upgrade process.


P.S. it would be great if you could avoid top-posting when
interacting with the BTS.

[Andrea Bolognani]

I think I'm starting to figure out what went wrong.

Looking at the log you shared, we can see this sequence of events:

Removing libvirt-daemon-system (9.0.0-4+deb12u1) ...

Preparing to unpack .../libvirt-daemon-driver-qemu_10.7.0-3_amd64.deb ...
Unpacking libvirt-daemon-driver-qemu (10.7.0-3) over (9.0.0-4+deb12u1) ...

Setting up libvirt-daemon-driver-qemu (10.7.0-3) ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ...
Configuration file '/etc/libvirt/qemu.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
Installing new version of config file /etc/libvirt/qemu.conf ...

Selecting previously unselected package libvirt-daemon-system.
Preparing to unpack .../libvirt-daemon-system_10.7.0-3_amd64.deb ...
Preparing transfer of config file /etc/apparmor.d/abstractions/libvirt-qemu (from libvirt-daemon-system to libvirt-daemon-driver-qemu) ...
Preparing transfer of config file /etc/libvirt/qemu.conf (from libvirt-daemon-system to libvirt-daemon-driver-qemu) ...
Unpacking libvirt-daemon-system (10.7.0-3) ...
Setting up libvirt-daemon-system (10.7.0-3) ...

For the transfer process to work correctly, we need preinst for the
package that conffiles are being transferred from (in this case
daemon-system) to run before preinst for the package they're being
transferred to (in this case daemon-driver-qemu).

This is supposed to be enforced by the following relationships
included in debian/control:

Package: libvirt-daemon-driver-qemu
Breaks: libvirt-daemon-system (<< 10.6.0-2~)
Replaces: libvirt-daemon-system (<< 10.6.0-2~)

However, as can be seen above, apt decides against configuring
daemon-system before other packages and removes it instead. As a
consequence, the transfer process breaks down.

I need to figure out how to ensure that the various steps happen in
the expected order. I have a couple of ideas, but some
experimentation will be necessary.

Going by the logs, you seem to have Xfce installed on your test
environment. Anything notable about the setup? I'm hoping that by
creating an installation as similar as possible to yours I will be
able to trigger the same upgrade failure... That'd make fixing it
easier.

[Kevin Otte]

Post by Andrea Bolognani
Going by the logs, you seem to have Xfce installed on your test
environment. Anything notable about the setup? I'm hoping that by
creating an installation as similar as possible to yours I will be
able to trigger the same upgrade failure... That'd make fixing it
easier.

I primarily run Sway these days but still have Xfce installed from when
it was my daily driver. I don't know of anything else that would be
notable, but here's a dpkg --get-selections for reference:

[Andrea Bolognani]

Apologies for taking a while to get back to you.

I was hoping that recreating your setup as closely as possible would
allow me to reproduce the issue locally, but I have been completely
unsuccessful despite several attempts.

In the end, I just cheated by adding:

Package: libvirt-daemon-system
Pre-Depends: libvirt-daemon-driver-qemu (= ${binary:Version})

This forces APT to fully configure daemon-driver-qemu before
daemon-system, thus triggering the behavior you reported.

Another thing that I realized going over the output you shared is:
the fact that you're getting a prompt asking you how to deal with
qemu.conf is actually a good thing. The contents of that file have
changed significantly between bookworm and today, but our conffile
handling logic would just preserve the file as it exists on the
system, if it has been modified, instead of prompting the user.
That's clearly an inferior experience.

Overally, I'm becoming convinced that our best course of action might
be to stop trying to be clever, drop all the custom logic and let
dpkg go through its built-in handling of conffiles.

Based on the tests I've performed so far, there are only two
drawbacks to this approach:

* dpkg will still report the conffiles as belonging to the old
package, in addition to the new package, though they will be
marked as obsolete in the first case;

* conffiles that had been deleted by the user before the upgrade
might reappear.

The former is slighly annoying, but it seems that simply calling:

$ sudo apt reinstall libvirt-daemon-system [...]

after the upgrade is enough to make it go away. I haven't been able
to trigger the latter at all, though that might be due to some
mistake on my part.

Regardless, neither seems so severe that it would make the approach
nonviable. Documenting this in the release notes should be enough to
properly inform users that they need to pay a bit more attention to
libvirt this time around. I was planning to do that anyway, since the
package has changed so much since bookworm.

I'll run more tests tomorrow.

[Kevin Otte]

Post by Andrea Bolognani
Apologies for taking a while to get back to you.

No worries. The world does seem to be a bit hectic of late.

Post by Andrea Bolognani
I was hoping that recreating your setup as closely as possible would
allow me to reproduce the issue locally, but I have been completely
unsuccessful despite several attempts.

I do seem to excel at stumbling onto edge cases :)

Post by Andrea Bolognani
...
Documenting this in the release notes should be enough to
properly inform users that they need to pay a bit more attention to
libvirt this time around. I was planning to do that anyway, since the
package has changed so much since bookworm.

As an avid reader of the release notes, I think this will be sufficient
and appreciated.