Bug#789775: zabbix-server: Zabbix server exits after 'One child process died'

[Chris Jones]

Package: zabbix-server-pgsql
Version: 1:2.4.5+dfsg-1
File: zabbix-server
Severity: important

Dear Maintainer,

Since upgrading to 2.4.5 (testing) from 2.2.7 (from stable) the Zabbix
server stops after running for some time (2-3 days).

The log reports:

14353:20150624:031057.640 One child process died
(PID:14454,exitcode/signal:1). Exiting ...
14353:20150624:031059.665 syncing history data...
14353:20150624:031059.669 item
"50340458-6568-21d8-4ad7-5aebac14ecb0:vmware.vm.hv.name[{$URL},{HOST.HOST}]"
became not supported: Timeout was reached
14353:20150624:031059.669 item
"503456b4-4503-6d61-0420-ea590eaf76cc:vmware.vm.vfs.fs.size[{$URL},{HOST.HOST},C:\,total]"
became not supported: Timeout was reached
14353:20150624:031059.747 syncing history data done
14353:20150624:031059.747 syncing trends data...
14353:20150624:031101.978 syncing trends data done
14353:20150624:031101.999 Zabbix Server stopped. Zabbix 2.4.5 (revision
53282).

Stop messages in the current set of logs on the sytem:
14353:20150624:031057.640 One child process died
(PID:14454,exitcode/signal:1). Exiting ...
28294:20150621:191551.817 One child process died
(PID:28360,exitcode/signal:1). Exiting ...
18948:20150618:215546.024 One child process died
(PID:19013,exitcode/signal:1). Exiting ...

-- System Information:
Debian Release: 8.1
APT prefers stable
APT policy: (700, 'stable'), (250, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)

Versions of packages zabbix-server-pgsql depends on:
ii adduser 3.113+nmu3
ii fping 3.10-2
ii libc6 2.19-18
ii libcurl3-gnutls 7.38.0-4+deb8u2
ii libiksemel3 1.4-2
ii libldap-2.4-2 2.4.40+dfsg-1
ii libodbc1 2.3.1-3
ii libopenipmi0 2.0.16-1.4
ii libpq5 9.4.3-0+deb8u1
ii libsnmp30 5.7.2.1+dfsg-1
ii libssh2-1 1.4.3-4.1
ii libxml2 2.9.1+dfsg1-5
ii lsb-base 4.1+Debian13+nmu1
ii ucf 3.0030

Versions of packages zabbix-server-pgsql recommends:
ii postgresql 9.4+165
ii snmpd 5.7.2.1+dfsg-1

Versions of packages zabbix-server-pgsql suggests:
ii logrotate 3.8.7-1+b1
ii snmp-mibs-downloader 1.1
ii zabbix-frontend-php 1:2.4.5+dfsg-1

-- Configuration Files:
/etc/default/zabbix-server 6a53dba77db32cb0615efdffaac54dfe [Errno 2] No
such file or directory: u'/etc/default/zabbix-server
6a53dba77db32cb0615efdffaac54dfe'

-- no debconf information

[Dmitry Smirnov]

Post by Chris Jones
Since upgrading to 2.4.5 (testing) from 2.2.7 (from stable) the Zabbix
server stops after running for some time (2-3 days).

Although I'm running zabbix-server-mysql for a while, I've never had this
problem with 2.4.5. I'm not sure how to troubleshoot it. Could it be a problem
in database? If so would you be able to VACUUM Zabbix tables or perhaps try to
back-up Zabbix database to make sure it works?

Also what init system are you using? Zabbix-server Systemd .service file
contains "Restart=on-abnormal" which should be able to restart server
automatically...

[Dmitry Smirnov]

Hi Chris,

Two more thoughts -- it may be useful to increase log verbosity and check
syslog as well. The latter may answer if zabbix server segfaults or geting
OOM-killed...

[Chris Jones]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Dmitry Smirnov
Two more thoughts -- it may be useful to increase log verbosity and check
syslog as well. The latter may answer if zabbix server segfaults or geting
OOM-killed...

Nothing relevant in syslog.

I've upped the debug level hopefull that will catch something if/when it
dies next.

- --
Chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVi+7rAAoJEIOwkgET13LrCmEQAJjeqGH+T6V4xuG1M0UqTGRz
EVclMTf2ncWmF6Ejj+PMnLW2chH9kv7+rD2cnVjv3eeY5FP2Bm7sOaYn8wJAqHVa
kPHf+U9YzhWHePLPu7FWyN1Dn/pPQSYtRfeqEv4FLV4/ZR/EiEAQJ7OJbyOZDezT
M400PqhhLvn3aZRYm6aWta2otCN+32ILYar5JtdSgxz0DsTTXOcam2YBLwuVCqwV
y/IQKmo++f2tXRM9MlVs/8aAX68hUEeoOJpwkPe1FwKcu+BglHF9FuJSwBt1Ft+c
u4BPo5zXLC2SXYQR5ENhY3yy2kXnrMjGj+agvZZJEc5Fp2x99fWPgYI965iNXxvl
8fW9mM2gyBjAgDeo5WY0Wx+tYfMJIJSL7kpHu4uetbKHZv5GNGsPiziHOB1RZB4A
xEKF+uqwk5GzRoy3PjqvbBEEKmi92Z/6c08bn/vOMX03t+bq4sVMwIYqHZlB6KyX
aLYNsfOOkmf4jcLVcoG62Fk6FDX0Wy9arOmZdW5K86N1sjKXpGgJhN7mX3OmAnbB
5s6jC1BVyff3vF+RVAvGKwPQbqiGsi3SghxBv57XYphoIdYTJSMh+wiR1ygZ/VSn
a6A9qTx/QZP/VzEGn9jDMHZUr255PdUvtWSVw5mstrYjmkEc0X640eV5s+AHA36T
E3HCSrU8WnFFeBZJT373
=Of0u
-----END PGP SIGNATURE-----

[Chris Jones]

Post by Dmitry Smirnov

Post by Dmitry Smirnov
Two more thoughts -- it may be useful to increase log verbosity and check
syslog as well. The latter may answer if zabbix server segfaults or

geting

Post by Dmitry Smirnov
OOM-killed...

Nothing relevant in syslog.
I've upped the debug level hopefull that will catch something if/when it
dies next.

So with a higher debug level the server failed to keep up with the load,
but didn’t die.

I've had to turn the debug level back down. Its failed again.

The logs says:

27964:20150708:004244.746 syncing history data done
27964:20150708:004244.746 syncing trends data...
27964:20150708:004302.928 syncing trends data done
27964:20150708:004302.943 Zabbix Server stopped. Zabbix 2.4.5 (revision
53282).

SystemD says:

# service zabbix-server status
● zabbix-server.service - Zabbix Server (PostgreSQL)
Loaded: loaded (/lib/systemd/system/zabbix-server.service; disabled)
Active: inactive (dead)
Docs: man:zabbix_server

Jun 25 12:41:30 zabbix.chi.swan.ac.uk zabbix_server[6328]: zabbix_server
[6328]: command sent successfully
Jun 25 12:46:51 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service:
Supervising process 6433 which is not our child. We'll most likely not
notice when it exits.
Jul 01 16:01:01 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service
stop-sigterm timed out. Killing.
Jul 01 16:01:01 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service:
main process exited, code=killed, status=9/KILL
Jul 01 16:01:01 zabbix.chi.swan.ac.uk systemd[1]: Unit
zabbix-server.service entered failed state.
Jul 01 16:01:29 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service:
Supervising process 5245 which is not our child. We'll most likely not
notice when it exits.
Jul 01 16:04:59 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service
stop-sigterm timed out. Killing.
Jul 01 16:04:59 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service:
main process exited, code=killed, status=9/KILL
Jul 01 16:04:59 zabbix.chi.swan.ac.uk systemd[1]: Unit
zabbix-server.service entered failed state.
Jul 06 11:06:22 zabbix.chi.swan.ac.uk systemd[1]: zabbix-server.service:
Supervising process 27964 which is not our child. We'll most likely not
notice when it exits.

--
Chris

[Dmitry Smirnov]

Post by Chris Jones
27964:20150708:004244.746 syncing history data done
27964:20150708:004244.746 syncing trends data...
27964:20150708:004302.928 syncing trends data done
27964:20150708:004302.943 Zabbix Server stopped. Zabbix 2.4.5 (revision
53282).

It gives me very little to work with... Strangely enough it doesn't look like
Zabbix server is crashed -- it looks more like if it just exited...

Post by Chris Jones
# service zabbix-server status
● zabbix-server.service - Zabbix Server (PostgreSQL)
Loaded: loaded (/lib/systemd/system/zabbix-server.service; disabled)
Active: inactive (dead)

Why is your service disabled? Could you enable it please so hopefully systemd
(re)start it when needed?

Thanks.

[Chris Jones]

Post by Dmitry Smirnov
Why is your service disabled? Could you enable it please so hopefully systemd
(re)start it when needed?

No Idea... I enabled it, it died agian

This time systemd says:

# service zabbix-server status
● zabbix-server.service - Zabbix Server (PostgreSQL)
Loaded: loaded (/lib/systemd/system/zabbix-server.service; enabled)
Active: inactive (dead) since Tue 2015-07-14 00:39:36 BST; 1 day 10h ago
Docs: man:zabbix_server
Main PID: 21502 (code=exited, status=0/SUCCESS)

And the log says :

21502:20150714:003929.414 One child process died
(PID:21550,exitcode/signal:1). Exiting ...
21502:20150714:003931.438 syncing history data...
21502:20150714:003931.501 syncing history data done
21502:20150714:003931.501 syncing trends data...
21502:20150714:003935.707 syncing trends data done
21502:20150714:003935.818 Zabbix Server stopped. Zabbix 2.4.5 (revision
53282).

--
Chris

[Chris Jones]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Dmitry Smirnov

Post by Chris Jones
Since upgrading to 2.4.5 (testing) from 2.2.7 (from stable) the Zabbix
server stops after running for some time (2-3 days).

Although I'm running zabbix-server-mysql for a while, I've never had this
problem with 2.4.5. I'm not sure how to troubleshoot it. Could it be a problem
in database? If so would you be able to VACUUM Zabbix tables or perhaps try to
back-up Zabbix database to make sure it works?
Also what init system are you using? Zabbix-server Systemd .service file
contains "Restart=on-abnormal" which should be able to restart server
automatically...

I using systemd, next time it dies I'll pass on what systemd says about it.

A backup (using pg_dump) completed.

- --
Chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVi9FYAAoJEIOwkgET13LrXBAP/0vZxaSIUlk2X8AE0f+tfyI6
6enGGakXxPL9OgVZu7UelcCQU1KAiFm7Pv1wswjm1I8WZK5rzX6bGsZvQKqJE6Bs
gbA6PzKiWqcehKX+qeCMW51f+ki1fY2EF6DsBnV3uMTg3TSOP0Kd0WQbQi5i7iJZ
X31bU5/MbTxyIsXDY5Vf5Na+d8JOPISqhQptDjMiYSv780LW9jHtekb/PmUSfBoT
Is0PR4wbWyU8akcJrqmgrxOe1xGXf3rsX2EWcWWnfrn7QqdrtvcLEJvqJdZ4e8mh
3xbjNp3cLi4btSQnCYX5yjPjuqLlMpTw4FG0Quu/m6VyBjWv52Hf5PlmgzxmOoih
mS/MqDH/+aafGujcmVsFk0pQ40j+VXnjoYUBGJ8QhKuHzAQ26M1E3mqD9wR5mKrE
yNsFUqY22TEfZPsuAuv1j6LrTgm9FMy5fKX3F5OzhxfMxQBWXZxpcSQScGufoC4L
PMK6cEv1YemIFOqYtoVk9Nt+GkB5oKp/tuLuQLnbakhvikUH4Ol4eoBKc9DM7Bzj
ACR7AOvKcSJL/YGKnpX/IOIEHPNO3qhG4kmDdR2is/lrGOb8ET8LbOGsuQ0+A69N
EJGeFThaiB8EAwnNia1jTvhFI7gTfnMNA2ZRAZhlCnYwO37hSbwgejCMHY6gylP1
8hwqAvh4tu1xhL64xstB
=q+Q+
-----END PGP SIGNATURE-----