Bug#961594: Connection failed [IP: 151.101.112.204 80]

[Christoph Berg]

Package: mirrors,apt
Severity: normal

On the apt.postgresql.org buildds, I've repeatedly seen failures when
retrieving files from security.debian.org. It started a few weeks ago,
and stopped when I reported the problem on #debian-mirrors on the 20th
when jcristau modified some SRV records for the originating network.
(The problem was seen from a Hetzner host in 78.46.0.0/15 aka
HETZNER-RZ-NBG-BLK5).

Now the problem is back, this time from Hetzner 144.76.0.0/16
HETZNER-RZ-BLK-ERX1.

26 11:58 <Myon> 11:53:34 Err http://security.debian.org/debian-security stretch/updates/main
amd64 libldap-common all 2.4.44+dfsg-5+deb9u4
26 11:58 <Myon> 11:53:34 Connection failed [IP: 151.101.112.204 80]
26 11:58 <Myon> jcristau: it happened again
26 11:59 <Myon> this time from 144.76.0.0/16 HETZNER-RZ-BLK-ERX1
26 12:18 <jcristau> hmm there's 2 places apt seems to say "Connection failed", one is closed
connection while getting headers, i can't quite figure out the other one
26 12:19 <jcristau> Myon: can you file a bug against mirrors,apt so we can get input from the
apt folks? i'm not sure i can usefully escalate to fastly just yet
26 14:46 <Myon> if a bug with these two lines is enough, sure
26 14:46 <Myon> curious why it's always that .deb, bad cache?
26 14:47 <Myon> or maybe one bad backend?
26 14:49 <jcristau> yeah i don't know :/

Christoph

[Julien Cristau]

Post by Christoph Berg
Package: mirrors,apt
Severity: normal
On the apt.postgresql.org buildds, I've repeatedly seen failures when
retrieving files from security.debian.org. It started a few weeks ago,
and stopped when I reported the problem on #debian-mirrors on the 20th
when jcristau modified some SRV records for the originating network.
(The problem was seen from a Hetzner host in 78.46.0.0/15 aka
HETZNER-RZ-NBG-BLK5).
Now the problem is back, this time from Hetzner 144.76.0.0/16
HETZNER-RZ-BLK-ERX1.
26 11:58 <Myon> 11:53:34 Err http://security.debian.org/debian-security stretch/updates/main
amd64 libldap-common all 2.4.44+dfsg-5+deb9u4
26 11:58 <Myon> 11:53:34 Connection failed [IP: 151.101.112.204 80]
26 11:58 <Myon> jcristau: it happened again
26 11:59 <Myon> this time from 144.76.0.0/16 HETZNER-RZ-BLK-ERX1
26 12:18 <jcristau> hmm there's 2 places apt seems to say "Connection failed", one is closed
connection while getting headers, i can't quite figure out the other one
26 12:19 <jcristau> Myon: can you file a bug against mirrors,apt so we can get input from the
apt folks? i'm not sure i can usefully escalate to fastly just yet
26 14:46 <Myon> if a bug with these two lines is enough, sure
26 14:46 <Myon> curious why it's always that .deb, bad cache?
26 14:47 <Myon> or maybe one bad backend?
26 14:49 <jcristau> yeah i don't know :/

I wonder if turning on apt's Debug::Acquire::http would give more of a
clue on where things go wrong? OTOH given this is highly intermittent
it'd be quite noisy... Christoph, would you be able to give that a try?

Cheers,
Julien

[Christoph Berg]

Post by Julien Cristau

Post by Christoph Berg
26 11:58 <Myon> 11:53:34 Err http://security.debian.org/debian-security stretch/updates/main
amd64 libldap-common all 2.4.44+dfsg-5+deb9u4
26 11:58 <Myon> 11:53:34 Connection failed [IP: 151.101.112.204 80]

It just happened again, this time on jessie (for the first time I
think):

12:05:42 E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/k/krb5/libkrb5support0_1.12.1+dfsg-19+deb8u5_amd64.deb Connection failed [IP: 151.101.64.204 80]

Source IP in range 176.9.80.32 - 176.9.80.63
netname: HETZNER-fsn1-dc6

Post by Julien Cristau
I wonder if turning on apt's Debug::Acquire::http would give more of a
clue on where things go wrong? OTOH given this is highly intermittent
it'd be quite noisy... Christoph, would you be able to give that a try?

I'll do that now. The first two retries with that setting didn't
reproduce the problem, though.

Christoph

[Christoph Berg]

Post by Christoph Berg

Post by Julien Cristau
I wonder if turning on apt's Debug::Acquire::http would give more of a
clue on where things go wrong? OTOH given this is highly intermittent
it'd be quite noisy... Christoph, would you be able to give that a try?

I'll do that now. The first two retries with that setting didn't
reproduce the problem, though.

20:20:00 Get: 31 http://security.debian.org/debian-security stretch/updates/main amd64 libldap-2.4-2 amd64 2.4.44+dfsg-5+deb9u4 [219 kB]
20:22:05 GET /debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44%2bdfsg-5%2bdeb9u4_amd64.deb HTTP/1.1
20:22:05 Host: security.debian.org
20:22:05 User-Agent: Debian APT-HTTP/1.3 (1.4.10)
20:22:05
20:22:05
20:22:05 Answer for: http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44+dfsg-5+deb9u4_amd64.deb
20:22:05 HTTP/1.1 200 OK
20:22:05 Server: Apache
20:22:05 X-Content-Type-Options: nosniff
20:22:05 X-Frame-Options: sameorigin
20:22:05 Referrer-Policy: no-referrer
20:22:05 X-Xss-Protection: 1
20:22:05 Last-Modified: Thu, 23 Apr 2020 05:40:59 GMT
20:22:05 ETag: "35840-5a3eeb18b3cf9"
20:22:05 Cache-Control: public, max-age=2592000
20:22:05 Expires: Tue, 28 Apr 2020 19:09:10 GMT
20:22:05 X-Clacks-Overhead: GNU Terry Pratchett
20:22:05 Content-Type: application/x-debian-package
20:22:05 Via: 1.1 varnish
20:22:05 Content-Length: 219200
20:22:05 Accept-Ranges: bytes
20:22:05 Date: Wed, 03 Jun 2020 18:22:05 GMT
20:22:05 Via: 1.1 varnish
20:22:05 Age: 515696
20:22:05 Connection: keep-alive
20:22:05 X-Served-By: cache-fra19137-FRA, cache-hhn4026-HHN
20:22:05 X-Cache: HIT, HIT
20:22:05 X-Cache-Hits: 1, 1
20:22:05 X-Timer: S1591208526.784738,VS0,VE0
20:22:05
20:22:05 Get: 32 http://security.debian.org/debian-security stretch/updates/main amd64 libldap-2.4-2 amd64 2.4.44+dfsg-5+deb9u4 [219 kB]
20:24:10 GET /debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44%2bdfsg-5%2bdeb9u4_amd64.deb HTTP/1.1
20:24:10 Host: security.debian.org
20:24:10 User-Agent: Debian APT-HTTP/1.3 (1.4.10)
20:24:10
20:24:10
20:24:10 Answer for: http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44+dfsg-5+deb9u4_amd64.deb
20:24:10 HTTP/1.1 200 OK
20:24:10 Server: Apache
20:24:10 X-Content-Type-Options: nosniff
20:24:10 X-Frame-Options: sameorigin
20:24:10 Referrer-Policy: no-referrer
20:24:10 X-Xss-Protection: 1
20:24:10 Last-Modified: Thu, 23 Apr 2020 05:40:59 GMT
20:24:10 ETag: "35840-5a3eeb18b3cf9"
20:24:10 Cache-Control: public, max-age=2592000
20:24:10 Expires: Tue, 28 Apr 2020 19:09:10 GMT
20:24:10 X-Clacks-Overhead: GNU Terry Pratchett
20:24:10 Content-Type: application/x-debian-package
20:24:10 Via: 1.1 varnish
20:24:10 Content-Length: 219200
20:24:10 Accept-Ranges: bytes
20:24:10 Date: Wed, 03 Jun 2020 18:24:10 GMT
20:24:10 Via: 1.1 varnish
20:24:10 Age: 515821
20:24:10 Connection: keep-alive
20:24:10 X-Served-By: cache-fra19137-FRA, cache-hhn4074-HHN
20:24:10 X-Cache: HIT, HIT
20:24:10 X-Cache-Hits: 1, 2
20:24:10 X-Timer: S1591208651.836599,VS0,VE0
20:24:10
20:24:10 Get: 33 http://security.debian.org/debian-security stretch/updates/main amd64 libldap-2.4-2 amd64 2.4.44+dfsg-5+deb9u4 [219 kB]
20:24:10 Fetched 16.6 MB in 8min 30s (32.4 kB/s)
20:24:10 E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-common_2.4.44+dfsg-5+deb9u4_all.deb: Connection failed [IP: 151.101.112.204 80]
20:24:10 E: Unable to fetch some packages; try '-o APT::Get::Fix-Missing=true' to continue with missing packages
20:24:11 Reading package lists...

I wonder if the 2min delay before the 2nd last package points at
something. Possibly the transfer was ok for that .deb, but then apt
tries http keepalive but that's already closed?

It could be that the NAT layer in the build chroots here have bad
iptables rules that break this (they have isolated network namespaces
using newpid/newnet). But then, why does it only happen for
security.d.o only, and only for jessie+stretch when buster has also
security? It's also restricted to a set of VMs at Hetzner, while other
machines are fine.

Also, the phenomenon is new (~3 months old or so), while the (buster)
buildhosts are much older and the config hasn't been touched except
for kernel updates.

Christoph