Discussion:
Bug#934246: munin-node doesn't start any more after upgrade to buster
(too old to reply)
Paolo Benvenuto
2019-08-08 17:40:01 UTC
Permalink
Package: munin-node
Version: 2.0.49-1
Severity: important

Dear Maintainer,

I had munin and munin-node working well on stretch, after upgrading to buster munin-node service doesn't start any more.

I tried purging the package and reinstalling it, nothing changed.

$ sudo service munin-node status
● munin-node.service - Munin Node
Loaded: loaded (/lib/systemd/system/munin-node.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-08-08 17:47:43 CEST; 1h 47min ago
Docs: man:munin-node(1)
http://munin.readthedocs.org/en/stable-2.0/reference/munin-node.html
Process: 20465 ExecStart=/usr/sbin/munin-node $DAEMON_ARGS (code=exited, status=2)

ago 08 17:47:43 catho2 systemd[1]: munin-node.service: Service RestartSec=100ms expired, scheduling restart.
ago 08 17:47:43 catho2 systemd[1]: munin-node.service: Scheduled restart job, restart counter is at 5.
ago 08 17:47:43 catho2 systemd[1]: Stopped Munin Node.
ago 08 17:47:43 catho2 systemd[1]: munin-node.service: Start request repeated too quickly.
ago 08 17:47:43 catho2 systemd[1]: munin-node.service: Failed with result 'exit-code'.
ago 08 17:47:43 catho2 systemd[1]: Failed to start Munin Node.



-- System Information:
Debian Release: 10.0
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-5-amd64 (SMP w/12 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8), LANGUAGE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages munin-node depends on:
ii libnet-server-perl 2.009-1
ii lsb-base 10.2019051400
ii munin-common 2.0.49-1
ii munin-plugins-core 2.0.49-1
ii netbase 5.6
ii perl 5.28.1-6

Versions of packages munin-node recommends:
ii gawk 1:4.2.1+dfsg-1
ii munin-plugins-extra 2.0.49-1
ii procps 2:3.3.15-2

Versions of packages munin-node suggests:
ii munin 2.0.49-1
pn munin-plugins-java <none>

-- Configuration Files:
/etc/munin/plugin-conf.d/README [Errno 13] Permesso negato: '/etc/munin/plugin-conf.d/README'
/etc/munin/plugin-conf.d/munin-node [Errno 13] Permesso negato: '/etc/munin/plugin-conf.d/munin-node'

-- no debconf inf
d***@sumpfralle.de
2019-08-08 21:40:01 UTC
Permalink
Hello Paolo,

thank you for your bug report!


Am Thu, 08 Aug 2019 19:36:08 +0200
Post by Paolo Benvenuto
I had munin and munin-node working well on stretch, after upgrading to buster
munin-node service doesn't start any more.
That sounds unpleasant.

Did you already find a reason for this behaviour?
What is the result of running "munin-node" directly (as root)?

Cheers,
Lars
Paolo Benvenuto
2019-08-09 14:20:01 UTC
Permalink
Post by Paolo Benvenuto
Post by Paolo Benvenuto
I had munin and munin-node working well on stretch, after upgrading to
buster
Post by Paolo Benvenuto
munin-node service doesn't start any more.
That sounds unpleasant.
Did you already find a reason for this behaviour?
no
Post by Paolo Benvenuto
What is the result of running "munin-node" directly (as root)?
# munin-node
doesn't produce any error

after that:

$ ps ax|grep munin
12471 ? Ss 0:00 /usr/bin/perl -wT /usr/sbin/munin-node

and it seems that something more is executed every 5 minutes.
Post by Paolo Benvenuto
Cheers,
Lars
Paolo Benvenuto
2019-08-09 15:20:01 UTC
Permalink
now the graphs are produced normally.

It seems that is a problem with the server start script

Il giorno ven 9 ago 2019 alle ore 16:12 Paolo Benvenuto <
Post by Paolo Benvenuto
Post by Paolo Benvenuto
Post by Paolo Benvenuto
I had munin and munin-node working well on stretch, after upgrading to
buster
Post by Paolo Benvenuto
munin-node service doesn't start any more.
That sounds unpleasant.
Did you already find a reason for this behaviour?
no
Post by Paolo Benvenuto
What is the result of running "munin-node" directly (as root)?
# munin-node
doesn't produce any error
$ ps ax|grep munin
12471 ? Ss 0:00 /usr/bin/perl -wT /usr/sbin/munin-node
and it seems that something more is executed every 5 minutes.
Post by Paolo Benvenuto
Cheers,
Lars
d***@sumpfralle.de
2019-08-10 11:50:01 UTC
Permalink
Hallo Paolo,


Am Fri, 9 Aug 2019 16:12:17 +0200
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
What is the result of running "munin-node" directly (as root)?
# munin-node
doesn't produce any error
$ ps ax|grep munin
12471 ? Ss 0:00 /usr/bin/perl -wT /usr/sbin/munin-node
This is weird. The systemd service should have done the same.
Post by Paolo Benvenuto
and it seems that something more is executed every 5 minutes.
This is a either a cron job (on the munin master) or the executed plugins on a
node (being requested by the master).

I still do not understand, the (trivial) systemd service should fail.

Did you take a look at /var/log/munin/munin-node.log?
(scrolling back to the time before your manual start)

Do you have any other idea for debugging the service on your host?
Otherwise I would suggest something quite low-level:
add "strace -o /tmp/munin-node-service.log -f" in front of the "ExecStart" in
the munin-node service file
(/etc/systemd/system/multi-user.target.wants/munin-node.service).

Cheers,
Lars
Paolo Benvenuto
2019-08-10 14:20:02 UTC
Permalink
Post by d***@sumpfralle.de
I still do not understand, the (trivial) systemd service should fail.
Did you take a look at /var/log/munin/munin-node.log?
(scrolling back to the time before your manual start)
I don't see anything significant.
I killed munin-node, and then restarted the service. The log says:

2019/08/10-16:03:15 Munin::Node::Server (type Net::Server::Fork) starting!
pid(4573)
Resolved [*]:4949 to [::]:4949, IPv6
Not including resolved host [0.0.0.0] IPv4 because it will be handled by
[::] IPv6
Binding to TCP port 4949 on host :: with IPv6

ipv6??????????

fromr:

sudo systemctl start munin-node.service ; journalctl -xe

I get:

ago 10 16:07:48 catho2 systemd[1]: Starting Munin Node...
-- Subject: L'unità munin-node.service inizia la fase di avvio
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- L'unità munin-node.service ha iniziato la fase di avvio.
ago 10 16:07:48 catho2 munin-node[10019]: Couldn't open log file
"/var/log/munin/munin-node.log" [No such file or directory]. at
/usr/share/perl5/Net/Server.pm line 209.
ago 10 16:07:48 catho2 systemd[1]: munin-node.service: Control process
exited, code=exited, status=2/INVALIDARGUMENT

systemd[1]: munin-node.service: Control process exited, code=exited,
status=2/INVALIDARGUMENT

and many other tries to start
Post by d***@sumpfralle.de
Do you have any other idea for debugging the service on your host?
add "strace -o /tmp/munin-node-service.log -f" in front of the "ExecStart" in
the munin-node service file
(/etc/systemd/system/multi-user.target.wants/munin-node.service).
Cheers,
Lars
d***@sumpfralle.de
2019-08-10 20:00:02 UTC
Permalink
Hello Paolo,


Am Sat, 10 Aug 2019 16:10:38 +0200
Post by Paolo Benvenuto
2019/08/10-16:03:15 Munin::Node::Server (type Net::Server::Fork) starting!
pid(4573)
Resolved [*]:4949 to [::]:4949, IPv6
Not including resolved host [0.0.0.0] IPv4 because it will be handled by
[::] IPv6
Binding to TCP port 4949 on host :: with IPv6
ipv6??????????
This is fine: munin-node supports IPv4 and IPv6.
Binding to an IPv6 port by default also means binding to the respective IPv4
port (see "/proc/sys/net/ipv6/bindv6only").
Post by Paolo Benvenuto
sudo systemctl start munin-node.service ; journalctl -xe
ago 10 16:07:48 catho2 systemd[1]: Starting Munin Node...
-- Subject: L'unità munin-node.service inizia la fase di avvio
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- L'unità munin-node.service ha iniziato la fase di avvio.
ago 10 16:07:48 catho2 munin-node[10019]: Couldn't open log file
"/var/log/munin/munin-node.log" [No such file or directory]. at
/usr/share/perl5/Net/Server.pm line 209.
ago 10 16:07:48 catho2 systemd[1]: munin-node.service: Control process
exited, code=exited, status=2/INVALIDARGUMENT
The problem with creating /var/log/munin/munin-node.log seems to be the culprit.
The directory should habe been created by the systemd component "munin.tmpfile"
during bootup.
I am confused, that this directory is missing on your host.

Do you have an idea?

Cheers,
Lars
Paolo Benvenuto
2019-08-10 20:30:01 UTC
Permalink
Post by d***@sumpfralle.de
Post by Paolo Benvenuto
sudo systemctl start munin-node.service ; journalctl -xe
ago 10 16:07:48 catho2 systemd[1]: Starting Munin Node...
-- Subject: L'unità munin-node.service inizia la fase di avvio
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- L'unità munin-node.service ha iniziato la fase di avvio.
ago 10 16:07:48 catho2 munin-node[10019]: Couldn't open log file
"/var/log/munin/munin-node.log" [No such file or directory]. at
/usr/share/perl5/Net/Server.pm line 209.
ago 10 16:07:48 catho2 systemd[1]: munin-node.service: Control process
exited, code=exited, status=2/INVALIDARGUMENT
The problem with creating /var/log/munin/munin-node.log seems to be the culprit.
The directory should habe been created by the systemd component "munin.tmpfile"
during bootup.
I am confused, that this directory is missing on your host.
The only thing I saw: /var/log/munin/munin-node.log had root:root owner, I
changed to munin:adm, but the service fails the same way.

Do you have an idea?
Post by d***@sumpfralle.de
Cheers,
Lars
d***@sumpfralle.de
2019-08-10 22:20:01 UTC
Permalink
Hello Paolo,


Am Sat, 10 Aug 2019 22:21:42 +0200
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
The problem with creating /var/log/munin/munin-node.log seems to be the culprit.
The directory should habe been created by the systemd component "munin.tmpfile"
during bootup.
I am confused, that this directory is missing on your host.
The only thing I saw: /var/log/munin/munin-node.log had root:root owner, I
changed to munin:adm, but the service fails the same way.
ok - the directory exists - thus the problem seems to be an access issue when
the service is started via systemd.
I could imagine an issue with any kind of sandboxing configured on your host.

Maybe for a start: take a look at
/etc/systemd/system/multi-user.target.wants/munin-node.service. There you can
see the following execution parameters:
PrivateDevices=false
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
Please comment out all of them and check, whether the service still fails.
(the above change modifies a file of the package - not a conffile - thus the
change will get lost with the next package upgrade)

If this fixes the issue, then please find out, which of the setting is causing
the failure.

If this does not fix the issue, then maybe you could think of any kind of
sandboxing security feature, that you configured for this host or its
virtualization provider? (selinux, apparmor, ...)

Cheers,
Lars
Paolo Benvenuto
2019-08-11 08:20:01 UTC
Permalink
Post by d***@sumpfralle.de
Hello Paolo,
Am Sat, 10 Aug 2019 22:21:42 +0200
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
The problem with creating /var/log/munin/munin-node.log seems to be the culprit.
The directory should habe been created by the systemd component "munin.tmpfile"
during bootup.
I am confused, that this directory is missing on your host.
The only thing I saw: /var/log/munin/munin-node.log had root:root owner,
I
Post by Paolo Benvenuto
changed to munin:adm, but the service fails the same way.
ok - the directory exists - thus the problem seems to be an access issue when
the service is started via systemd.
I could imagine an issue with any kind of sandboxing configured on your host.
What is a sandboxing? I am running a server, without any chroot or anything
similar
Post by d***@sumpfralle.de
Maybe for a start: take a look at
/etc/systemd/system/multi-user.target.wants/munin-node.service. There you can
PrivateDevices=false
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
Please comment out all of them and check, whether the service still fails.
(the above change modifies a file of the package - not a conffile - thus the
change will get lost with the next package upgrade)
I commented them out, now the service starts!
Post by d***@sumpfralle.de
If this fixes the issue, then please find out, which of the setting is causing
the failure.
how?!?!
d***@sumpfralle.de
2019-08-11 17:00:01 UTC
Permalink
Hello Paolo,

thank for helping us finding the root cause of this problem!


Am Sun, 11 Aug 2019 10:12:36 +0200
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
I could imagine an issue with any kind of sandboxing configured on your host.
What is a sandboxing? I am running a server, without any chroot or anything
similar
Maybe "sandboxing" is not a very precise term. I wanted to refer to software,
that restricts programs in any way. SELinux and apparmor are popular examples.
Also the "seccomp" (Ithink) features configured in our systemd services
definitions are such kind of sandboxing methods. Virtualization (like docker)
can also enforce some restrictions on running processes.
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
Maybe for a start: take a look at
/etc/systemd/system/multi-user.target.wants/munin-node.service. There you can
PrivateDevices=false
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
Please comment out all of them and check, whether the service still fails.
(the above change modifies a file of the package - not a conffile - thus
the change will get lost with the next package upgrade)
I commented them out, now the service starts!
Great - this is good to know!
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
If this fixes the issue, then please find out, which of the setting is
causing the failure.
how?!?!
You can stop the service again, remove some of the comments and try to start it
again, until you find the minimal set of options, that need to be disabled for
you.
Afterwards we (munin packagers) can try to find out, under which circumstances
this specific restriction could cause a problem.

At the moment I suspect, that /var/log/munin is somehow symlinked on your
system into /tmp or /run. But this is just a very wild guess.

Thank you for your help!

Cheers,
Lars
Paolo Benvenuto
2019-08-11 19:30:01 UTC
Permalink
Post by d***@sumpfralle.de
At the moment I suspect, that /var/log/munin is somehow symlinked on your
system into /tmp or /run. But this is just a very wild guess.
actually /var/log is a symlink to /home/log
Would it be the reason of the problem?
Post by d***@sumpfralle.de
Thank you for your help!
Cheers,
Lars
d***@sumpfralle.de
2019-08-11 20:20:01 UTC
Permalink
Hello Paolo,


Am Sun, 11 Aug 2019 21:18:58 +0200
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
At the moment I suspect, that /var/log/munin is somehow symlinked on your
system into /tmp or /run. But this is just a very wild guess.
actually /var/log is a symlink to /home/log
Would it be the reason of the problem?
yes, "PrivateTmp=true" would create a separate (empty) /tmp directory in the
namespace of the munin-node process (see "man systemd.service").
Thus if you symlinked /var/log/munin to /tmp/munin-log (or something similar),
then /tmp/munin-log would not exist for the new munin-node process.

Are you using such a setup? In this case I would recommend to mount a tmpfs
onto /var/log/munin or /var/log instead. The munin service files supports such a
read-only customization without changes.

Cheers,
Lars
Stig Sandbeck Mathisen
2019-08-12 05:20:01 UTC
Permalink
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
At the moment I suspect, that /var/log/munin is somehow symlinked on your
system into /tmp or /run. But this is just a very wild guess.
actually /var/log is a symlink to /home/log
Would it be the reason of the problem?
The ProtectHome= setting will make /home look empty (or readonly) for a
service. The symlink of /var/log to /home/log will therefore point to a
nonexistant location for that service by default.

Look at man:systemd.exec(5) for more information about that setting.
--
Stig Sandbeck Mathisen
Paolo Benvenuto
2019-08-17 20:00:02 UTC
Permalink
I put again /var/log as a directory, and everything works perfectly.

The bug can be closed

don Paolo Benvenuto


Il giorno lun 12 ago 2019 alle ore 07:05 Stig Sandbeck Mathisen <
Post by Stig Sandbeck Mathisen
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
At the moment I suspect, that /var/log/munin is somehow symlinked on
your
Post by Paolo Benvenuto
Post by d***@sumpfralle.de
system into /tmp or /run. But this is just a very wild guess.
actually /var/log is a symlink to /home/log
Would it be the reason of the problem?
The ProtectHome= setting will make /home look empty (or readonly) for a
service. The symlink of /var/log to /home/log will therefore point to a
nonexistant location for that service by default.
Look at man:systemd.exec(5) for more information about that setting.
--
Stig Sandbeck Mathisen
Loading...