Discussion:
Bug#943962: mariadb-server-10.3: mysqld crashes and hangs, no longer processing requests
(too old to reply)
Richard van den Berg
2019-11-01 21:00:02 UTC
Permalink
Package: mariadb-server-10.3
Version: 1:10.3.17-0+deb10u1
Severity: important

I run mysqldump (through automysqlbackup) daily. Several times per week
during this backup mysqld hangs. The process however stays running and still
accepts TCP and socket connections, however no SQL queries are ever
answered anymore. This is very serious because systemd does not catch this
and leaves the process running. Only a "pkill -9 mysqld" can resolve the
situation.

In the 15+ years I have been using mysql/mariadb I have never encoutered
this situation where my system is left hanging without a working database.
If mysqld crashed (this is bad enough) the process should end itself so
systemd can restart it.

The messages in error.log are:

corrupted size vs. prev_size
191101 7:23:41 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.17-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=5
max_threads=153
thread_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467422
K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f2b600014b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2bc4814dd8 thread_stack 0x49000


-- System Information:
Debian Release: 10.1
APT prefers stable
APT policy: (990, 'stable'), (900, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mariadb-server-10.3 depends on:
ii adduser 3.118
ii debconf [debconf-2.0] 1.5.71
ii galera-3 25.3.25-2
ii gawk 1:4.2.1+dfsg-1
ii iproute2 4.20.0-2
ii libc6 2.28-10
ii libdbi-perl 1.642-1+b1
ii libgnutls30 3.6.7-4
ii libpam0g 1.3.1-5
ii libstdc++6 8.3.0-6
ii lsb-base 10.2019051400
ii lsof 4.91+dfsg-1
ii mariadb-client-10.3 1:10.3.17-0+deb10u1
ii mariadb-common 1:10.3.17-0+deb10u1
ii mariadb-server-core-10.3 1:10.3.17-0+deb10u1
ii passwd 1:4.5-1.1
ii perl 5.28.1-6
ii psmisc 23.2-1
ii rsync 3.1.3-6
ii socat 1.7.3.2-2
ii zlib1g 1:1.2.11.dfsg-1

Versions of packages mariadb-server-10.3 recommends:
ii libhtml-template-perl 2.97-1

Versions of packages mariadb-server-10.3 suggests:
ii bsd-mailx [mailx] 8.1.2-0.20180807cvs-1
ii mailx 1:20071201-3
pn mariadb-test <none>
pn netcat-openbsd <none>
pn tinyca <none>

-- debconf information excluded
Otto Kekäläinen
2019-11-01 21:30:02 UTC
Permalink
Hello!

Did you report this bug upstream (as the output said "To report this
bug, see https://mariadb.com/kb/en/reporting-bugs"). This is unlikely
related to the packaging done in Debian.


pe 1. marrask. 2019 klo 22.57 Richard van den Berg
Post by Richard van den Berg
Package: mariadb-server-10.3
Version: 1:10.3.17-0+deb10u1
Severity: important
I run mysqldump (through automysqlbackup) daily. Several times per week
during this backup mysqld hangs. The process however stays running and still
accepts TCP and socket connections, however no SQL queries are ever
answered anymore. This is very serious because systemd does not catch this
and leaves the process running. Only a "pkill -9 mysqld" can resolve the
situation.
In the 15+ years I have been using mysql/mariadb I have never encoutered
this situation where my system is left hanging without a working database.
If mysqld crashed (this is bad enough) the process should end itself so
systemd can restart it.
corrupted size vs. prev_size
191101 7:23:41 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.3.17-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=5
max_threads=153
thread_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467422
K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f2b600014b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2bc4814dd8 thread_stack 0x49000
Debian Release: 10.1
APT prefers stable
APT policy: (990, 'stable'), (900, 'oldstable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
ii adduser 3.118
ii debconf [debconf-2.0] 1.5.71
ii galera-3 25.3.25-2
ii gawk 1:4.2.1+dfsg-1
ii iproute2 4.20.0-2
ii libc6 2.28-10
ii libdbi-perl 1.642-1+b1
ii libgnutls30 3.6.7-4
ii libpam0g 1.3.1-5
ii libstdc++6 8.3.0-6
ii lsb-base 10.2019051400
ii lsof 4.91+dfsg-1
ii mariadb-client-10.3 1:10.3.17-0+deb10u1
ii mariadb-common 1:10.3.17-0+deb10u1
ii mariadb-server-core-10.3 1:10.3.17-0+deb10u1
ii passwd 1:4.5-1.1
ii perl 5.28.1-6
ii psmisc 23.2-1
ii rsync 3.1.3-6
ii socat 1.7.3.2-2
ii zlib1g 1:1.2.11.dfsg-1
ii libhtml-template-perl 2.97-1
ii bsd-mailx [mailx] 8.1.2-0.20180807cvs-1
ii mailx 1:20071201-3
pn mariadb-test <none>
pn netcat-openbsd <none>
pn tinyca <none>
-- debconf information excluded
_______________________________________________
pkg-mysql-maint mailing list
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-mysql-maint
--
- Otto
Richard van den Berg
2019-11-01 23:30:01 UTC
Permalink
Post by Otto Kekäläinen
Did you report this bug upstream (as the output said "To report this
bug, see https://mariadb.com/kb/en/reporting-bugs"). This is unlikely
related to the packaging done in Debian.
I did not report this upstream yet. The proper thing to do with these
type of crashes is to provide a gdb trace. Unfortunately I am not able
to find dbg versions of the mariadb-server-10.3 and
mariadb-server-core-10.3 debian packages. Are they available somewhere?

Kind regards,

Richard
Otto Kekäläinen
2019-11-02 12:10:01 UTC
Permalink
Hello!

Yes, upstream developers will value a good trace.

Debian ships debug symbols for all packages by default. You just need
to install them from the debug repository. Maybe this helps
https://wiki.debian.org/HowToGetABacktrace ?
Richard van den Berg
2019-11-06 14:40:01 UTC
Permalink
Any pointers to where I can find mariadb-server-core-10.3 for buster with debug symbols would be
appreciated.
Richard van den Berg
2019-11-06 15:20:01 UTC
Permalink
I found the debug packages in http://debug.mirrors.debian.org/debian-debug/pool/main/m/mariadb-10.3/

It seems I broke my apt preferences so it could not find them automatically.

Running mysqld now with a gdb on standby...
Richard van den Berg
2019-11-08 09:50:02 UTC
Permalink
I was able to get a stack trace and reported this upsteam at https://jira.mariadb.org/browse/MDEV-21010
Loading...