Discussion:
Bug#868650: network-online.target should not be reached before getaddrinfo() can succeed
(too old to reply)
Wouter Verhelst
2017-07-17 09:00:04 UTC
Permalink
Raw Message
Package: systemd
Version: 232-23
Severity: normal
File: network-online.target

Hi,

After fiddling with ***@.service in the nbd-client package so that it looks like this:

[Unit]
Description=NBD client connection for %i
Documentation=man:nbd-client
PartOf=nbd.service
Before=dev-%i.device
After=network-online.target
DefaultDependencies=no
Conflicts=shutdown.target
[Service]
Type=forking
ExecStart=/sbin/nbd-client %i
[Install$
RequiredBy=dev-%i.device
RequiredBy=dev-%ip1.device
RequiredBy=dev-%ip2.device
[... repeat the above for all pX, X = [1, 15] ...]

and rebooting a machine with a fully configured nbd setup, I see the following:

***@testvm:~# systemctl status ***@nbd0
● ***@nbd0.service - NBD client connection for nbd0
Loaded: loaded (/etc/systemd/system/***@.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2017-07-16 11:54:16 CEST; 4min 22s ago
Docs: man:nbd-client
Process: 366 ExecStart=/sbin/nbd-client nbd0 (code=exited, status=1/FAILURE)

jul 16 11:54:16 nbdclient systemd[1]: Starting NBD client connection for nbd0...
jul 16 11:54:16 nbdclient nbd-client[366]: getaddrinfo failed: Address family for hostname not supported
jul 16 11:54:16 nbdclient systemd[1]: ***@nbd0.service: Control process exited, code=exited status=1
jul 16 11:54:16 nbdclient systemd[1]: Failed to start NBD client connection for nbd0.
jul 16 11:54:16 nbdclient systemd[1]: ***@nbd0.service: Unit entered failed state.
jul 16 11:54:16 nbdclient systemd[1]: ***@nbd0.service: Failed with result 'exit-code'.

This is after just having booted and logged on as root. If I immediately run
"systemctl restart ***@nbd0" at this point, everything works well.

It would seem that systemd reaches network-online.target before the network is
fully functional.

[The below is from the machine on which I did the above, a slightly
outdated jessie VM, which was installed before the release. If
necessary, I can try this with a more up-to-date version of jessie,
testing, or unstable, but I'm on an airplane right now...]

-- Package-specific info:

-- System Information:
Debian Release: 9.0
APT prefers testing-updates
APT policy: (500, 'testing-updates'), (500, 'testing')
Architecture: amd64
(x86_64)

Kernel: Linux 4.9.0-3-amd64 (SMP w/1 CPU core)
Locale: LANG=nl_BE.UTF-8, LC_CTYPE=nl_BE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages systemd depends on:
ii adduser 3.115
ii libacl1 2.2.52-3+b1
ii libapparmor1 2.11.0-3
ii libaudit1 1:2.6.7-2
ii libblkid1 2.29.2-1
ii libc6 2.24-10
ii libcap2 1:2.25-1
ii libcryptsetup4 2:1.7.3-4
ii libgcrypt20 1.7.6-1
ii libgpg-error0 1.26-2
ii libidn11 1.33-1
ii libip4tc0 1.6.0+snapshot20161117-6
ii libkmod2 23-2
ii liblz4-1 0.0~r131-2+b1
ii liblzma5 5.2.2-1.2+b1
ii libmount1 2.29.2-1
ii libpam0g 1.1.8-3.5
ii libseccomp2 2.3.1-2.1
ii libselinux1 2.6-3+b1
ii libsystemd0 232-23
ii mount 2.29.2-1
ii util-linux 2.29.2-1

Versions of packages systemd recommends:
ii dbus 1.10.18-1
pn libpam-systemd <none>

Versions of packages systemd suggests:
pn policykit-1 <none>
pn systemd-container <none>
pn systemd-ui <none>

Versions of packages systemd is related to:
pn dracut <none>
ii initramfs-tools 0.130
ii udev
Michael Biebl
2017-07-17 10:20:02 UTC
Permalink
Raw Message
Control: tags -1 + moreinfo

Hi Wouter
Post by Wouter Verhelst
It would seem that systemd reaches network-online.target before the network is
fully functional
network-online.target is just a concept not an implementation. It
depends on the individual network config tool to hook properly into it.

What tools do you use to configure your network?

systemd-networkd has systemd-network-wait-online.service which needs to
be enabled and then hooks into network-online.target,

NetworkManager has NetworkManager-wait-online.service which needs to be
enabled and then hooks into network-online-target,

ifupdown has no wait-online service. It simply runs ifup -a before
network-online.target. This excludes allow-hotplug interfaces.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Wouter Verhelst
2017-07-17 11:10:04 UTC
Permalink
Raw Message
Hi,
Post by Michael Biebl
Control: tags -1 + moreinfo
Hi Wouter
Post by Wouter Verhelst
It would seem that systemd reaches network-online.target before the network is
fully functional
network-online.target is just a concept not an implementation. It
depends on the individual network config tool to hook properly into it.
Right, okay.
Post by Michael Biebl
What tools do you use to configure your network?
The test VM used ifupdown.
Post by Michael Biebl
systemd-networkd has systemd-network-wait-online.service which needs to
be enabled and then hooks into network-online.target,
NetworkManager has NetworkManager-wait-online.service which needs to be
enabled and then hooks into network-online-target,
ifupdown has no wait-online service. It simply runs ifup -a before
network-online.target. This excludes allow-hotplug interfaces.
I suppose that "ifup -a" returns before everything's set up properly
then; that would explain it.

Thanks,
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Michael Biebl
2017-07-17 11:20:02 UTC
Permalink
Raw Message
Control: reassign -1 ifupdown
Post by Wouter Verhelst
Hi,
Post by Michael Biebl
Control: tags -1 + moreinfo
Hi Wouter
Post by Wouter Verhelst
It would seem that systemd reaches network-online.target before the network is
fully functional
network-online.target is just a concept not an implementation. It
depends on the individual network config tool to hook properly into it.
Right, okay.
Post by Michael Biebl
What tools do you use to configure your network?
The test VM used ifupdown.
Post by Michael Biebl
systemd-networkd has systemd-network-wait-online.service which needs to
be enabled and then hooks into network-online.target,
NetworkManager has NetworkManager-wait-online.service which needs to be
enabled and then hooks into network-online-target,
ifupdown has no wait-online service. It simply runs ifup -a before
network-online.target. This excludes allow-hotplug interfaces.
I suppose that "ifup -a" returns before everything's set up properly
then; that would explain it.
I'm reassigning this to ifupdown then. Assuming you use "auto"
interfaces this is supposed to work.

Regards,
Michael
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Michael Biebl
2017-07-17 11:50:02 UTC
Permalink
Raw Message
Post by Michael Biebl
Post by Wouter Verhelst
I suppose that "ifup -a" returns before everything's set up properly
then; that would explain it.
I'm reassigning this to ifupdown then. Assuming you use "auto"
interfaces this is supposed to work.
Apparently I wasn't (sorry, missed that bit in your previous email).
After fixing that, it works.
Thanks for the help, closing the bug now (because there isn't one, after
all).
Might be good to have ifupdown deal with interfaces in a way so that
even with allow-hotplug (which the installer uses by default) at least
some network is up and functional when "network-online.target" is
reached; but other than that, we should be fine.
Supporting allow-hotplug interfaces for network-online.target is
something we discussed quite a while ago (unfortunately I can't find the
archive anymore).
I don't think we ever filed a bug report for this though, so we could
just reopen this one?

A long time ago I created a PoC for ifupdown-wait-online [1]. It was
rather simple though. I simply blocked until at least one interface was
configured. One would probably want something more sophisticated.

Michael


[1] https://people.debian.org/~biebl/ifupdown-wait-online.tar.gz
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Wouter Verhelst
2017-07-17 12:00:01 UTC
Permalink
Raw Message
Post by Michael Biebl
A long time ago I created a PoC for ifupdown-wait-online [1]. It was
rather simple though. I simply blocked until at least one interface was
configured. One would probably want something more sophisticated.
It's a good start though, and would probably work for 99% of cases.
Let's not make the perfect be the enemy of the good?
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Guus Sliepen
2017-07-17 14:10:01 UTC
Permalink
Raw Message
Post by Wouter Verhelst
Post by Michael Biebl
A long time ago I created a PoC for ifupdown-wait-online [1]. It was
rather simple though. I simply blocked until at least one interface was
configured. One would probably want something more sophisticated.
It's a good start though, and would probably work for 99% of cases.
Let's not make the perfect be the enemy of the good?
I don't know about that 99% value. The problem is usually seen
when mounting remote filesystems or when running some services that want
to bind to a specific IP address but can't be bothered to use
IP_FREEBIND. You want to wait for an external host to become reachable,
or a specific IP address to be assigned locally.

Waiting until at least one interface is configured is quite naive: it
will probably work very well when you only have one interface, but what
if you have multiple interfaces? In that case, I'd rather have it block
until all interfaces are up. This will still cover the single interface
situation.

In any case, I'll add the script.
--
Met vriendelijke groet / with kind regards,
Guus Sliepen <***@debian.org>
Michael Biebl
2017-07-17 17:20:01 UTC
Permalink
Raw Message
Post by Guus Sliepen
Post by Michael Biebl
A long time ago I created a PoC for ifupdown-wait-online [1]. It was
rather simple though. I simply blocked until at least one interface was
configured. One would probably want something more sophisticated.
Waiting until at least one interface is configured is quite naive: it
will probably work very well when you only have one interface, but what
if you have multiple interfaces? In that case, I'd rather have it block
until all interfaces are up. This will still cover the single interface
situation.
Indeed, the script is very naive. As said, it was meant as a PoC to show
how to hook into network-online.target.
Ideally this would be built directly into ifupdown instead of via hook
scripts. Maybe have a small ifup-wait-online binary which monitors the
state file via inotify or have ifup talk directly to this binary (e.g.
vai a socket). That binary could have different options:
- wait for all interfaces to be up
- wait for a single interface to be up
- wait for a given interface to be up
- wait for all auto interfaces, if there is no auto interface, wait for
at least one allow-hotplug interface
- some other criteria

Waiting for all interfaces to be up could have unpleasant side-effects.
E.g. I had a system with a auto eth0 and a allow-hotplug wlan config.
I did not always have the allow-hotplug device plugged in. So
ifup-wait-online would run into a time out every time.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Guus Sliepen
2017-07-17 19:40:03 UTC
Permalink
Raw Message
Post by Michael Biebl
- wait for all auto interfaces, if there is no auto interface, wait for
at least one allow-hotplug interface
I think that's a very good default heuristic for ifupdown-wait-online.
--
Met vriendelijke groet / with kind regards,
Guus Sliepen <***@debian.org>
Guus Sliepen
2017-07-21 23:30:02 UTC
Permalink
Raw Message
Post by Guus Sliepen
Post by Michael Biebl
- wait for all auto interfaces, if there is no auto interface, wait for
at least one allow-hotplug interface
I think that's a very good default heuristic for ifupdown-wait-online.
I also had a wait-online.sh script I made myself lying around. I merged
your functionality with mine. I've not made it depend on a if-up.d
script to signal an interface coming online, I'll optimize it later.

The script supports several ways to wait:

- Wait for ifup to bring something up (either a specific interface, or
if none is specified, either all auto interfaces, or one hotplug interface,
as discussed).
- Wait for a route to appear to a certain address, or a default gateway.
- Wait for a host to respond to ping packets.
- Don't wait.

You'll have to systemctl enable ifupdown-wait-online.service manually
for now, but if it works OK then I'll turn it on by default.
--
Met vriendelijke groet / with kind regards,
Guus Sliepen <***@debian.org>
Loading...