Bug#1090355: libvirt-daemon-driver-network: Switch of firewall backend to nftables breaks NAT for guest machines

[Max Hofer]

Package: libvirt-daemon-driver-network
Version: 10.10.0-3
Severity: normal

Upgrading to libvirt breaks the internett access to my guest machines
using NAT forwarding. Default firewalld is installed.

I attached the iptables rules from libvirt 10.10.0-1 (using iptables as
firewall backend) and the new one after the upgrade with the nftables as
backend.

Workaround: enable setting 'firewall_backend = "iptables"' in
/etc/libvirt/network.conf restores the old behavior.


-- System Information:
Debian Release: trixie/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.11.6-amd64 (SMP w/16 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libvirt-daemon-driver-network depends on:
ii dnsmasq-base [dnsmasq-base] 2.90-6
ii iptables 1.8.11-2
ii libc6 2.40-4
ii libgcc-s1 14.2.0-11
ii libglib2.0-0t64 2.82.4-1
ii libvirt-common 10.10.0-3
ii libvirt-daemon 10.10.0-3
ii libvirt0 10.10.0-3
ii nftables 1.1.1-1

libvirt-daemon-driver-network recommends no packages.

libvirt-daemon-driver-network suggests no packages.

-- no debconf information

[Andrea Bolognani]

Post by Max Hofer
Package: libvirt-daemon-driver-network
Version: 10.10.0-3
Severity: normal
Upgrading to libvirt breaks the internett access to my guest machines
using NAT forwarding. Default firewalld is installed.
I attached the iptables rules from libvirt 10.10.0-1 (using iptables as
firewall backend) and the new one after the upgrade with the nftables as
backend.
Workaround: enable setting 'firewall_backend = "iptables"' in
/etc/libvirt/network.conf restores the old behavior.

[...]

Post by Max Hofer
*filter
:INPUT ACCEPT [40677:4186813]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [41189:72181069]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:LIBVIRT_FWI - [0:0]
:LIBVIRT_FWO - [0:0]
:LIBVIRT_FWX - [0:0]
:LIBVIRT_INP - [0:0]
:LIBVIRT_OUT - [0:0]
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -j LIBVIRT_OUT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
COMMIT

Thanks for reaching out.

I'm no firewall expert but I see that there are some Docker rules in
there, so I think you might be hitting the same issue mentioned here:

https://fedoraproject.org/wiki/Changes/LibvirtVirtualNetworkNFTables#Known_issue:_docker

Can you try disabling Docker and checking whether the libvirt
nftables backend works as expected then?

We might need to document this incompatibility more prominently, for
example in the release notes.

[NoisyCoil]

Package: libvirt-daemon-driver-network
Version: 10.10.0-3
Followup-For: Bug #1090355
X-Debbugs-Cc: ***@tutanota.com

I see the same behavior by simply having ufw installed and enabled, no special
rules, no docker installed. Disabling ufw or manually adding blanket INPUT and
FORWARD rules to enable incoming and outgoing traffic from/to the virbr+
interfaces fixes this, but neither is a good solution.

It seems that libvirt should provide extra firewall rules if it wants to play
nicely with nftables. Having ufw (or docker, or anything else really) installed
should not prevent NAT from working. On the other hand, if for some reason this
is the new intended behavior, then the change should be documented together
with the precise list of rules needed to enable NAT when the default for INPUT
and FORWARD is DROP (i.e. usually whenever a firewall is active).

[Andrea Bolognani]

Post by NoisyCoil
I see the same behavior by simply having ufw installed and enabled, no special
rules, no docker installed. Disabling ufw or manually adding blanket INPUT and
FORWARD rules to enable incoming and outgoing traffic from/to the virbr+
interfaces fixes this, but neither is a good solution.
It seems that libvirt should provide extra firewall rules if it wants to play
nicely with nftables. Having ufw (or docker, or anything else really) installed
should not prevent NAT from working. On the other hand, if for some reason this
is the new intended behavior, then the change should be documented together
with the precise list of rules needed to enable NAT when the default for INPUT
and FORWARD is DROP (i.e. usually whenever a firewall is active).

This too is a known issue:

https://fedoraproject.org/wiki/Changes/LibvirtVirtualNetworkNFTables#Known_issue:_non-firewalld_firewall_mgmt_tools

Both this and the Docker incompatibility are probably fine in the
context of a distro such as Fedora, where firewalld and Podman are
the "blessed" solutions in their respective fields, but Debian is
much less opinionated than that.

I need to spend some more time thinking about this, but switching the
default network backend back to iptables might be the most reasonable
solution.

[Andrea Bolognani]

[re-added the bug report]

You were right. It is the combination of Docker with libvirt using
`nftables` as firewall backend which breaks NAT.
I could reproduce it by using the default firewall backend, removing
docker.io, reboot machine, restarted the default network with `sudo virsh
net-start default` --> guest system could access internet.
Interestingly, with this setup, no firewall rules are created at all and NAT
is still working.
Reinstalling docker.io, keeping nftables as firewall backedn --> guest
system could not access the internet.
`iptables -nL` doesn't show any LIBVIRT_* chains.
Changing firewall backend to iptables, reboot, restart libvirt network -->
guest system could access internet. NAT chains LIBVIRT_* are created.

When using the nftables backend, the rules will not be created using
iptables so that tool won't know about them.

If you run "nftables list ruleset" you should see a bunch of
libvirt-related rules.

Note that iptables is using the same kernel API as nftables these
days, at least unless you go out of your way to tell it not to:

$ update-alternatives --query iptables
Name: iptables
Link: /usr/sbin/iptables
Slaves:
iptables-restore /usr/sbin/iptables-restore
iptables-save /usr/sbin/iptables-save
Status: auto
Best: /usr/sbin/iptables-nft
Value: /usr/sbin/iptables-nft

Alternative: /usr/sbin/iptables-legacy
Priority: 10
Slaves:
iptables-restore /usr/sbin/iptables-legacy-restore
iptables-save /usr/sbin/iptables-legacy-save

Alternative: /usr/sbin/iptables-nft
Priority: 20
Slaves:
iptables-restore /usr/sbin/iptables-nft-restore
iptables-save /usr/sbin/iptables-nft-save

I think it would be a good idea to document it somewhere, since the
combination of libvirt+docker.io installation is not uncommon.

I agree.

[NoisyCoil]

Package: libvirt-daemon-driver-network
Version: 10.10.0-3
Followup-For: Bug #1090355

Post by Andrea Bolognani
https://fedoraproject.org/wiki/Changes/LibvirtVirtualNetworkNFTables#Known_issue:_non-firewalld_firewall_mgmt_tools

Confirmed. DHCP not working is how I first learned about this issue, and the
behavior I see is that described in [1]. With respect to distros, one of the

Post by Andrea Bolognani
The immediate workaround is for anyone who uses UFW to tell libvirt to switch
back to its iptables backend again. If UFW is Arch's default firewall tool,
then Arch builds of libvirt should be made to set iptables as the default.

Debian has no default firewall, but ufw's popcon is 20990 vs firewalld's 5010.


[1] https://gitlab.com/libvirt/libvirt/-/issues/644

[Andrea Bolognani]

Post by NoisyCoil

Post by Andrea Bolognani
https://fedoraproject.org/wiki/Changes/LibvirtVirtualNetworkNFTables#Known_issue:_non-firewalld_firewall_mgmt_tools

Confirmed. DHCP not working is how I first learned about this issue, and the
behavior I see is that described in [1]. With respect to distros, one of the

Post by Andrea Bolognani
The immediate workaround is for anyone who uses UFW to tell libvirt to switch
back to its iptables backend again. If UFW is Arch's default firewall tool,
then Arch builds of libvirt should be made to set iptables as the default.

Debian has no default firewall, but ufw's popcon is 20990 vs firewalld's 5010.

Additional data point. ufw is installed and enabled by default on
Ubuntu server. Docker is one of the documented alternatives among
container runtimes; Podman isn't.

I think it's obvious at this point that the default should go back to
iptables. Users can easily switch to nftables if desired, but things
should work reasonably well out of the box.

I'll prepare a patch and an upload over the next few days.

[n***@tutanota.com]

Thanks Andrea!