Discussion:
Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
(too old to reply)
James Addison
2020-05-03 10:00:01 UTC
Permalink
Package: apt
Version: 1.8.2
Severity: normal

Dear Maintainer,

During installation of packages under Debian Buster[1], I've encountered a
repeatable (non-TLS) HTTP download error that occurs during download of
openjdk-11-jdk-headless.

It's possible this may expose a rare edge case in apt's HTTP client, and/or
it may be a transient network condition; either way it seems worth reporting.

The steps to reproduce the problem are:

1. Install GPG

apt-get update && apt-get install -y gpg

2. Approve apt GPG keys for snapshot.debian.org

gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 5E61B217265DA9807A23C5FF4DFAB270CAA96DFA
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 80D15823B7FD1561F9F7BCDDDC30D7C23CBBABEE
gpg -a --export 5E61B217265DA9807A23C5FF4DFAB270CAA96DFA | apt-key add -
gpg -a --export 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517 | apt-key add -
gpg -a --export 80D15823B7FD1561F9F7BCDDDC30D7C23CBBABEE | apt-key add -

NB: gpg will fail to import, with a 'new key but contains no user ID - skipped'
message if the '--keyserver' option is omitted.

3. Configure retrieval of snapshot packages

export SNAPSHOT="20200502T085134Z"
rm /etc/apt/sources.list
printf "deb http://snapshot.debian.org/archive/debian/${SNAPSHOT}/ buster main\n" >> /etc/apt/sources.list
printf "deb http://snapshot.debian.org/archive/debian-security/${SNAPSHOT}/ buster/updates main\n" >> /etc/apt/sources.list
printf "deb http://snapshot.debian.org/archive/debian/${SNAPSHOT}/ buster-updates main\n" >> /etc/apt/sources.list

4. Attempt installation of openjdk-11-jdk

apt-get update && apt-get install -y openjdk-11-jdk


Expected Results:

Successful installation of packages.


Actual Results:

The following output appears during processing of the command:

Get:185 http://snapshot.debian.org/archive/debian/20200502T085134Z buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB]
Get:186 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jre amd64 11.0.7+10-3~deb10u1 [34.3 kB]
Get:187 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Err:187 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 193.62.202.27 80]
Get:188 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk amd64 11.0.7+10-3~deb10u1 [2607 kB]
Get:189 http://snapshot.debian.org/archive/debian/20200502T085134Z buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
Get:190 http://snapshot.debian.org/archive/debian/20200502T085134Z buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB]


Anecdotally, after retrying this process a number of times yesterday, my ISP
appeared to introduce some rate limiting (unfortunately I didn't record any
statistics).

During the slower/limited network condition, I was not able to
reproduce the problem.

Restrictions appear to have been lifted today and the problem reliably
appears each time I attempt the installation.


Related Issues:

I have tried applying a workaround for a similar-sounding issue[2] by
adding '-o Acquire::http::Pipeline-Depth="0"' to the apt-get install
command, but this does not resolve the issue.

Thank you,
James

[1] - https://hub.docker.com/layers/debian/library/debian/buster-slim/images/sha256-ed8df275d4736ad00e69c4f58c2d9aa3f917911be47753e6d8d25368bcafa38d

[2] - https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1801338

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Sandbox "";
APT::Sandbox::User "_apt";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-modules";
APT::VersionedKernelPackages:: "linux-modules-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "linux-image-unsigned";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
APT::VersionedKernelPackages:: "linux-modules-.*";
APT::VersionedKernelPackages:: "linux-tools";
APT::VersionedKernelPackages:: "linux-cloud-tools";
APT::VersionedKernelPackages:: "linux-buildinfo";
APT::VersionedKernelPackages:: "linux-source";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "contrib/metapackages";
APT::Never-MarkAuto-Sections:: "non-free/metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Move-Autobit-Sections "";
APT::Move-Autobit-Sections:: "oldlibs";
APT::Move-Autobit-Sections:: "contrib/oldlibs";
APT::Move-Autobit-Sections:: "non-free/oldlibs";
APT::Move-Autobit-Sections:: "restricted/oldlibs";
APT::Move-Autobit-Sections:: "universe/oldlibs";
APT::Move-Autobit-Sections:: "multiverse/oldlibs";
APT::AutoRemove "";
APT::AutoRemove::SuggestsImportant "false";
APT::Update "";
APT::Update::Post-Invoke "";
APT::Update::Post-Invoke:: "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Architectures:: "i386";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "0";
APT::Compressor::zstd "";
APT::Compressor::zstd::Name "zstd";
APT::Compressor::zstd::Extension ".zst";
APT::Compressor::zstd::Binary "false";
APT::Compressor::zstd::Cost "60";
APT::Compressor::lz4 "";
APT::Compressor::lz4::Name "lz4";
APT::Compressor::lz4::Extension ".lz4";
APT::Compressor::lz4::Binary "false";
APT::Compressor::lz4::Cost "50";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "100";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-6n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "200";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "300";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-6";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "400";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-6";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "";
Dir::Cache::pkgcache "";
Dir::Etc "etc/apt";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::netrcparts "auth.conf.d";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::planners "";
Dir::Bin::planners:: "/usr/lib/apt/planners";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::gzip "/bin/gzip";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lz4 "/usr/bin/lz4";
Dir::Bin::zstd "/usr/bin/zstd";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Log::Planner "eipp.log.xz";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.ucf-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::AllowInsecureRepositories "0";
Acquire::AllowWeakRepositories "0";
Acquire::AllowDowngradeToInsecureRepositories "0";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::IndexTargets "";
Acquire::IndexTargets::deb "";
Acquire::IndexTargets::deb::Packages "";
Acquire::IndexTargets::deb::Packages::MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
Acquire::IndexTargets::deb::Packages::flatMetaKey "Packages";
Acquire::IndexTargets::deb::Packages::ShortDescription "Packages";
Acquire::IndexTargets::deb::Packages::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
Acquire::IndexTargets::deb::Packages::flatDescription "$(RELEASE) Packages";
Acquire::IndexTargets::deb::Packages::Optional "0";
Acquire::IndexTargets::deb::Translations "";
Acquire::IndexTargets::deb::Translations::MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatMetaKey "$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::ShortDescription "Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatDescription "$(RELEASE) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb-src "";
Acquire::IndexTargets::deb-src::Sources "";
Acquire::IndexTargets::deb-src::Sources::MetaKey "$(COMPONENT)/source/Sources";
Acquire::IndexTargets::deb-src::Sources::flatMetaKey "Sources";
Acquire::IndexTargets::deb-src::Sources::ShortDescription "Sources";
Acquire::IndexTargets::deb-src::Sources::Description "$(RELEASE)/$(COMPONENT) Sources";
Acquire::IndexTargets::deb-src::Sources::flatDescription "$(RELEASE) Sources";
Acquire::IndexTargets::deb-src::Sources::Optional "0";
Acquire::Changelogs "";
Acquire::Changelogs::URI "";
Acquire::Changelogs::URI::Origin "";
Acquire::Changelogs::URI::Origin::Debian "https://metadata.ftp-master.debian.org/changelogs/@***@_changelog";
Acquire::Changelogs::URI::Origin::Ubuntu "https://changelogs.ubuntu.com/changelogs/pool/@CHANGEPATH@/changelog";
Acquire::Changelogs::AlwaysOnline "";
Acquire::Changelogs::AlwaysOnline::Origin "";
Acquire::Changelogs::AlwaysOnline::Origin::Ubuntu "1";
Acquire::GzipIndexes "true";
Acquire::Languages "";
Acquire::Languages:: "none";
Acquire::CompressionTypes "";
Acquire::CompressionTypes::xz "xz";
Acquire::CompressionTypes::bz2 "bzip2";
Acquire::CompressionTypes::lzma "lzma";
Acquire::CompressionTypes::gz "gzip";
Acquire::CompressionTypes::lz4 "lz4";
Acquire::CompressionTypes::zst "zstd";
DPkg "";
DPkg::Path "/usr/sbin:/usr/bin:/sbin:/bin";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true";
Binary "apt-config";
Binary::apt "";
Binary::apt::APT "";
Binary::apt::APT::Color "1";
Binary::apt::APT::Cache "";
Binary::apt::APT::Cache::Show "";
Binary::apt::APT::Cache::Show::Version "2";
Binary::apt::APT::Cache::AllVersions "0";
Binary::apt::APT::Cache::ShowVirtuals "1";
Binary::apt::APT::Cache::Search "";
Binary::apt::APT::Cache::Search::Version "2";
Binary::apt::APT::Cache::ShowDependencyType "1";
Binary::apt::APT::Cache::ShowVersion "1";
Binary::apt::APT::Get "";
Binary::apt::APT::Get::Upgrade-Allow-New "1";
Binary::apt::APT::Get::Update "";
Binary::apt::APT::Get::Update::InteractiveReleaseInfoChanges "1";
Binary::apt::APT::Cmd "";
Binary::apt::APT::Cmd::Show-Update-Stats "1";
Binary::apt::APT::Keep-Downloaded-Packages "0";
Binary::apt::DPkg "";
Binary::apt::DPkg::Progress-Fancy "1";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- (no /etc/apt/preferences present) --


-- (no /etc/apt/preferences.d/* present) --


-- /etc/apt/sources.list --

deb http://snapshot.debian.org/archive/debian/20200502T085134Z/ buster main
deb http://snapshot.debian.org/archive/debian-security/20200502T085134Z/ buster/updates main
deb http://snapshot.debian.org/archive/debian/20200502T085134Z/ buster-updates main

-- (no /etc/apt/sources.list.d/* present) --


-- System Information:
Debian Release: 10.3
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.5.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: unable to detect

Versions of packages apt depends on:
ii adduser 3.118
ii debian-archive-keyring 2019.1
ii gpgv 2.2.12-1+deb10u1
ii libapt-pkg5.0 1.8.2
ii libc6 2.28-10
ii libgcc1 1:8.3.0-6
ii libgnutls30 3.6.7-4+deb10u3
ii libseccomp2 2.3.3-4
ii libstdc++6 8.3.0-6

Versions of packages apt recommends:
ii ca-certificates 20190110

Versions of packages apt suggests:
pn apt-doc <none>
pn aptitude | synaptic | wajig <none>
pn dpkg-dev <none>
ii gnupg 2.2.12-1+deb10u1
pn powermgmt-base <none>

-- no debconf information
James Addison
2020-05-03 13:50:02 UTC
Permalink
Package: snapshot.debian.org
Followup-For: Bug #959518

Dear Maintainer,

I've reassigned this issue to both 'apt' and 'snapshot.debian.org' since it
seems possible that either an HTTP client edge case and/or a server-side
connection-handling behaviour may be the cause(s).

It also seems possible that the rate-limiting mentioned previously is being
applied by the snapshot.debian.org server; that would be sensible given that
my client has retried download of packages repeatedly.

I'm not certain whether my bug tracker 'reassign' operation notified the
maintainers of the 'snapshot.debian.org' pseudo-package, so this update also
serves to (hopefully) ensure they're included on the thread.

Thanks again,
James

-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 5.5.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Julian Andres Klode
2020-08-04 08:20:01 UTC
Permalink
Post by James Addison
Package: apt
Version: 1.8.2
Severity: normal
Dear Maintainer,
During installation of packages under Debian Buster[1], I've encountered a
repeatable (non-TLS) HTTP download error that occurs during download of
openjdk-11-jdk-headless.
It's possible this may expose a rare edge case in apt's HTTP client, and/or
it may be a transient network condition; either way it seems worth reporting.
There are known bugs in the pipelining code that cause issues in an insignifcant
amount of cases, reproducing them is next to impossible. There's some kind of
race somewhere between the server traffic and the client.

I had a reproducer once for such an issue, but sadly, it did not last.

If you can still reproduce this, could you give the current master branch a try?
I went looking around for what *seems* wrong and fixed that, but uh, it might not
help at all.

and/or try the attached patch to get the location where it failed.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
James Addison
2020-08-04 22:40:02 UTC
Permalink
Package: apt,snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: ***@jp-hosting.net

Thanks for the update! - the issue *does* remain reproducible at the moment,
and unfortunately cherry-picking the fix (merge commit 7d22263) into the
1.8.2.z branch and building a version of apt from there doesn't seem to resolve
it.
James Addison
2020-08-04 23:20:01 UTC
Permalink
Package: apt,snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: ***@jp-hosting.net

FWIW, the series of steps in use for repro are:

# prereq: install apt compile-time dependencies
# prereq: add apt deb sources, keys as per docs at http://snapshot.debian.org/
$ git clone "https://salsa.debian.org/apt-team/apt.git"
$ cd apt
$ git checkout 1.8.2.z
$ git cherry-pick -m 1 7d222636954ec95382149e31b314e9828ba05a2e
$ cmake -DCMAKE_INSTALL_PREFIX=/ .
$ make
$ cmdline/apt -o Acquire::Check-Valid-Until=false update
$ cmdline/apt install -y openjdk-11-jdk

And the error manifests as:

Get:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1 [215 MB]
Err:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1
Undetermined Error [IP: 193.62.202.27 80]

If time allows it might be possible to prepare a self-contained Dockerfile
with the repro steps if that'd be useful for further debug/investigation.
David Kalnischkies
2020-08-05 06:30:01 UTC
Permalink
Post by James Addison
$ cmdline/apt -o Acquire::Check-Valid-Until=false update
$ cmdline/apt install -y openjdk-11-jdk
While you are using the built 'apt' in this way, apt is still talking to
the methods (including http) installed on your system, not the ones you
just built (and patched), so it would have been rather mysterious if it
had worked. ☺

Try:
$ cmdline/apt -o Acquire::Check-Valid-Until=false update -o dir::bin::methods=$(pwd)/methods
$ cmdline/apt install -y openjdk-11-jdk -o dir::bin::methods=$(pwd)/methods

(There are other things apt will still use the installed version of,
including config files and such, but that isn't important now)


Might be worthwhile to try the whole battery of the http changes Julian
did instead of just the single one as they interact with each other even
if they are somewhat independent.


Best regards

David Kalnischkies
Julian Andres Klode
2020-08-05 08:00:02 UTC
Permalink
Post by James Addison
Package: apt,snapshot.debian.org
Followup-For: Bug #959518
# prereq: install apt compile-time dependencies
# prereq: add apt deb sources, keys as per docs at http://snapshot.debian.org/
$ git clone "https://salsa.debian.org/apt-team/apt.git"
$ cd apt
$ git checkout 1.8.2.z
$ git cherry-pick -m 1 7d222636954ec95382149e31b314e9828ba05a2e
You also need the commit before that. This fixed the undetermined error
on top of the previous fixes, which probably moved the error into a different
code path :)

Especially

commit cb743d117bcc666dab4c5948b1227ed2edbd0578
Author: Julian Andres Klode <***@canonical.com>
Date: Mon Jun 29 11:45:45 2020 +0200

http: Only return false for EOF if we actually did not read anything

This should avoid the need to Flush the buffer in Die(), because
if we read anything, we are returning true, and not entering Die()
at that point.

Also Write() does not have a concept of EOF, so get rid of code
handling that there. Was that copied from Read()?

But probably the entire merge. These all act badly individually.

Really, just try the master branch instead of going cherry-picking individual
commits.
Post by James Addison
$ cmake -DCMAKE_INSTALL_PREFIX=/ .
$ make
$ cmdline/apt -o Acquire::Check-Valid-Until=false update
$ cmdline/apt install -y openjdk-11-jdk
This is wrong, as David already pointed out - it's missing -o Dir::Bin::Methods

I'd also build using dpkg-buildpackage so you get all the right flags. I can never
remember them all, but there's also stuff like libexec and so on that it passes.
Post by James Addison
Get:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1 [215 MB]
Err:159 http://snapshot.debian.org/archive/debian-security/20200801T030228Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.8+10-1~deb10u1
Undetermined Error [IP: 193.62.202.27 80]
If time allows it might be possible to prepare a self-contained Dockerfile
with the repro steps if that'd be useful for further debug/investigation.
No, that's not useful at all. You just put a sources.list and a trusted.gpg.d
in a directory (e.g. $PWD/x) with the other files, and then do

apt -o Dir=$PWD/x update
apt -o Dir=$PWD/x download openjdk-11-jdk-headless

I have attached x.tar for convenience.

And that reproduces the issue. Well at least it did early yesterday. Which
is why I could debug it and come up with a fix for it. Though, now i can't
reproduce the undetermined error without the fix either, I just get weird
other errors...

apt -o Dir=$PWD/x install openjdk-11-jdk -y -d

certainly did.

And then you can run this from a source tree like this (at least I do):

dpkg-buildpackage -b
./obj-x86_64-linux-gnu/cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/obj-x86_64-linux-gnu/methods update
./obj-x86_64-linux-gnu/cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/obj-x86_64-linux-gnu/methods update

You also want to pin snapshot.debian.org to the failing IP I think, as
the two servers are vastly different.


It's not super reproducible:

***@jak-t480s:~/Projects/Debian/apt:pu/http-fixes$ lxc exec bu apt download openjdk-11-jdk-headless
Get:1 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Err:1 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 2001:1af8:4020:b030:deb::185 80]
W: Download is performed unsandboxed as root as file '/root/openjdk-11-jdk-headless_11.0.7+10-3~deb10u1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: Failed to fetch http://snapshot.debian.org/archive/debian-security/20200502T085134Z/pool/updates/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.7+10-3~deb10u1_amd64.deb Undetermined Error [IP: 2001:1af8:4020:b030:deb::185 80]
***@jak-t480s:~/Projects/Debian/apt:pu/http-fixes$ lxc exec bu apt download openjdk-11-jdk-headless
Get:1 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Fetched 215 MB in 20s (10.9 MB/s)
W: Download is performed unsandboxed as root as file '/root/openjdk-11-jdk-headless_11.0.7+10-3~deb10u1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

Despite having pinned the IP in /etc/hosts.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
James Addison
2020-08-05 10:40:01 UTC
Permalink
Package: apt,snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: ***@jp-hosting.net

Thank you both! I should've done a more digging into the commit history for
the related HTTP fixes. I'm not sure I would've easily figured out the apt
method issue but it's a relief to see there's a command-line flag for that.

I'll try this again from master soon and glad to help with any other debug
(although as evidenced I'm a bit slow to figure out the various best practices
for testing all this).
Alex Thiessen
2020-08-25 09:10:02 UTC
Permalink
Dear maintainer,

I'm facing this issue (apt 2.1.10 (amd64)) and here are a couple
observations that might help you to grasp it better.

snapshot.d.o servers close the connection randomly, it took wget four
continuation retries to download the file completely. When called
again, the download is completed by wget flawlessly (might be a
transparent proxy kicking in, though).

logs:

----
$ wget http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
--2020-08-25 09:22:15--
http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27,
185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ...
Connecting to snapshot.debian.org
(snapshot.debian.org)|193.62.202.27|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 207236812 (198M)
Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’

openjdk-11-jdk-headless_11.0.3+ 9%[===>
] 17.97M 3.65MB/s in 5.2s

2020-08-25 09:22:21 (3.46 MB/s) - Connection closed at byte 18845696. Retrying.

--2020-08-25 09:22:22-- (try: 2)
http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
Connecting to snapshot.debian.org
(snapshot.debian.org)|193.62.202.27|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 207236812 (198M), 188391116 (180M) remaining
Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’

openjdk-11-jdk-headless_11.0.3+ 14%[++++===>
] 29.46M 3.45MB/s in 3.3s

2020-08-25 09:22:25 (3.45 MB/s) - Connection closed at byte 30896128. Retrying.

--2020-08-25 09:22:27-- (try: 3)
http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
Connecting to snapshot.debian.org
(snapshot.debian.org)|193.62.202.27|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 207236812 (198M), 176340684 (168M) remaining
Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’

openjdk-11-jdk-headless_11.0.3+ 20%[++++++++==>
] 40.87M 3.45MB/s in 3.3s

2020-08-25 09:22:31 (3.45 MB/s) - Connection closed at byte 42856448. Retrying.

--2020-08-25 09:22:34-- (try: 4)
http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
Connecting to snapshot.debian.org
(snapshot.debian.org)|193.62.202.27|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 207236812 (198M), 164380364 (157M) remaining
Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’

openjdk-11-jdk-headless_11.0.3+
100%[+++++++++++==========================================>] 197.64M
4.11MB/s in 40s

2020-08-25 09:23:15 (3.88 MB/s) -
‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ saved
[207236812/207236812]

$ rm openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
$ wget http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
--2020-08-25 09:25:38--
http://snapshot.debian.org/archive/debian/20190706T130900Z/pool/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb
Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27,
185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ...
Connecting to snapshot.debian.org
(snapshot.debian.org)|193.62.202.27|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 207236812 (198M)
Saving to: ‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’

openjdk-11-jdk-headless_11.0.3+
100%[=====================================================>] 197.64M
2.97MB/s in 54s

2020-08-25 09:26:32 (3.69 MB/s) -
‘openjdk-11-jdk-headless_11.0.3+7-5_amd64.deb’ saved
[207236812/207236812]
----

Best regards,
Alex Thiessen
James Addison
2020-09-14 16:30:02 UTC
Permalink
Package: snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: ***@jp-hosting.net

The issue appears reproducible at the moment with apt 1.8.2.1 compiled from source and the 'x.tar' configuration provided earlier.

# apt source directory, post-build
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y openjdk-11-jdk

...

Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 193.62.202.27 80]

This has occurred for a couple of different server IP addresses, including 185.17.185.185.
Julian Andres Klode
2020-09-14 17:00:02 UTC
Permalink
Post by James Addison
Package: snapshot.debian.org
Followup-For: Bug #959518
The issue appears reproducible at the moment with apt 1.8.2.1 compiled from source and the 'x.tar' configuration provided earlier.
# apt source directory, post-build
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y openjdk-11-jdk
...
Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 193.62.202.27 80]
This has occurred for a couple of different server IP addresses, including 185.17.185.185.
We only care about unstable for this bug. There is a whole bunch of
changes in http code and they won't be backported to stable releases.

Also, the previous comment by Alex Thiessen indicated that this is not a
bug in apt, but the server seems to close the connection, which means
there is nothing actionable here.

If you can produce an issue with the version of apt in unstable,
and it does not reproduce with wget or curl, please open a new
bug report for it.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Johannes Schauer
2020-09-14 19:40:01 UTC
Permalink
Hi,
Post by Julian Andres Klode
Post by James Addison
Package: snapshot.debian.org
Followup-For: Bug #959518
The issue appears reproducible at the moment with apt 1.8.2.1 compiled from source and the 'x.tar' configuration provided earlier.
# apt source directory, post-build
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y openjdk-11-jdk
...
Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 193.62.202.27 80]
This has occurred for a couple of different server IP addresses, including 185.17.185.185.
We only care about unstable for this bug. There is a whole bunch of
changes in http code and they won't be backported to stable releases.
Also, the previous comment by Alex Thiessen indicated that this is not a
bug in apt, but the server seems to close the connection, which means
there is nothing actionable here.
If you can produce an issue with the version of apt in unstable,
and it does not reproduce with wget or curl, please open a new bug report for
it.
I'm very familiar with snapshot.d.o from the client perspective. Julian is
correct, that it's the server closing the connection. But that doesn't mean
that it's not at least a wishlist bug or feature request in apt. Let me explain
a bit more.

For several projects (debrebuild, debbisect, buildprofile QA,
bootstrap.debian.net...) I regularly interact with snapshot.d.o. Doing this
plainly with apt is deemed to fail miserably with errors like:

# E: Failed to fetch [...] Error reading from server. Remote end closed connection
# E: Failed to fetch [...] Hash Sum mismatch
# E: Failed to fetch [...] Bad header line Bad header data
# Err:118 [...] Connection timed out

Yes, this is because of how snapshot.d.o throttles connections. For example
without additional measures, the following will fail:

$ curl http://snapshot.debian.org/archive/debian/20200909T084102Z/pool/main/q/qtwebengine-opensource-src/qtwebengine-opensource-src_5.14.2+dfsg1.orig.tar.xz >/dev/null
curl: (18) transfer closed with 217347024 bytes remaining to read

There are a couple of things that can be done to work around this problem when
using curl by adding options like:

--limit-rate=800k # this has the biggest effect
--retry 10 --retry-connrefused
--resolve snapshot.debian.org:80:193.62.202.27

But even those are not sufficient as snapshot.d.o will also cut the connection
early enough such that curl will fail with "network unreachable" which is not a
transient error, so curl will not retry establishing the connection.

The only thing that reliably worked for me with snapshot.d.o was the pycurl
based Python code at the end of this E-Mail. With that code, I can even
download for a full day reliably from snapshot.d.o without ever having hit the
Exception in the last line.

But as things stand, it is impossible to reliably use apt together with
snapshot.d.o. I'm not sure how to solve this problem. One way could surely be
to approach snapshot.d.o and ask them to somehow lift their very heavy
throttling policies. But another way to solve this problem would be to make apt
more resilient about mirrors with heavy throttling policies. I can think of
these wishlist bugs against apt:

- allow to specify a maximum bytes per second value for downloads (this has
the largest effect if set low enough)

- allow to set an option that makes apt automatically retry when a transient
error occurs

- allow to set custom resolve addresses for domains like done in my code below

I'm not saying that we shouldn't look into maybe making snapshot.d.o throttle
less, because as things stand, it's impossible to use it together with apt. But
there certainly also some things that apt can do and which will not only
benefit people working with snapshot.d.o but also people who are otherwise
using a mirror or proxy with heavy throttling.

Thanks!

cheers, josch





def download(url):
f = BytesIO()
maxretries = 10
for retrynum in range(maxretries):
try:
c = pycurl.Curl()
c.setopt(
c.URL, url,
)
# even 100 kB/s is too much sometimes
c.setopt(c.MAX_RECV_SPEED_LARGE, 800 * 1024) # bytes per second
c.setopt(c.CONNECTTIMEOUT, 30) # the default is 300
# sometimes, curl stalls forever and even ctrl+c doesn't work
start = time.time()
def progress(*data):
# a download must not last more than 5 minutes
# with 100 kB/s this means files cannot be larger than 31MB
if time.time() - start > 5 * 60:
print("transfer took too long")
return 0
c.setopt(pycurl.NOPROGRESS, 0)
c.setopt(pycurl.XFERINFOFUNCTION, progress)
# $ host snapshot.debian.org
# snapshot.debian.org has address 185.17.185.185
# snapshot.debian.org has address 193.62.202.27
# c.setopt(c.RESOLVE, ["snapshot.debian.org:80:185.17.185.185"])
if f.tell() != 0:
c.setopt(pycurl.RESUME_FROM, f.tell())
c.setopt(c.WRITEDATA, f)
c.perform()
assert c.getinfo(c.RESPONSE_CODE) in [200, 206], c.getinfo(c.RESPONSE_CODE)
c.close()
return f.getvalue()
except pycurl.error as e:
code, message = e.args
if code in [pycurl.E_PARTIAL_FILE, pycurl.E_COULDNT_CONNECT]:
if retrynum == maxretries - 1:
break
print("retrying...")
time.sleep(2 ** retrynum)
continue
else:
raise
raise Exception("failed too often...")
Julian Andres Klode
2020-09-14 20:00:01 UTC
Permalink
Post by Johannes Schauer
Hi,
Post by Julian Andres Klode
Post by James Addison
Package: snapshot.debian.org
Followup-For: Bug #959518
The issue appears reproducible at the moment with apt 1.8.2.1 compiled from source and the 'x.tar' configuration provided earlier.
# apt source directory, post-build
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods update && \
$ cmdline/apt -o Dir=$PWD/x -o Dir::Bin::Methods=$PWD/methods install -y openjdk-11-jdk
...
Get:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
Err:261 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
Undetermined Error [IP: 193.62.202.27 80]
This has occurred for a couple of different server IP addresses, including 185.17.185.185.
We only care about unstable for this bug. There is a whole bunch of
changes in http code and they won't be backported to stable releases.
Also, the previous comment by Alex Thiessen indicated that this is not a
bug in apt, but the server seems to close the connection, which means
there is nothing actionable here.
If you can produce an issue with the version of apt in unstable,
and it does not reproduce with wget or curl, please open a new bug report for
it.
I'm very familiar with snapshot.d.o from the client perspective. Julian is
correct, that it's the server closing the connection. But that doesn't mean
that it's not at least a wishlist bug or feature request in apt. Let me explain
a bit more.
For several projects (debrebuild, debbisect, buildprofile QA,
bootstrap.debian.net...) I regularly interact with snapshot.d.o. Doing this
# E: Failed to fetch [...] Error reading from server. Remote end closed connection
# E: Failed to fetch [...] Hash Sum mismatch
# E: Failed to fetch [...] Bad header line Bad header data
# Err:118 [...] Connection timed out
Yes, this is because of how snapshot.d.o throttles connections. For example
Frankly, it should throttle connections not kill them. Generally you do that
by refusing new GET requests or sending less data per time.
Post by Johannes Schauer
$ curl http://snapshot.debian.org/archive/debian/20200909T084102Z/pool/main/q/qtwebengine-opensource-src/qtwebengine-opensource-src_5.14.2+dfsg1.orig.tar.xz >/dev/null
curl: (18) transfer closed with 217347024 bytes remaining to read
There are a couple of things that can be done to work around this problem when
--limit-rate=800k # this has the biggest effect
--retry 10 --retry-connrefused
--resolve snapshot.debian.org:80:193.62.202.27
But even those are not sufficient as snapshot.d.o will also cut the connection
early enough such that curl will fail with "network unreachable" which is not a
transient error, so curl will not retry establishing the connection.
The only thing that reliably worked for me with snapshot.d.o was the pycurl
based Python code at the end of this E-Mail. With that code, I can even
download for a full day reliably from snapshot.d.o without ever having hit the
Exception in the last line.
But as things stand, it is impossible to reliably use apt together with
snapshot.d.o. I'm not sure how to solve this problem. One way could surely be
to approach snapshot.d.o and ask them to somehow lift their very heavy
throttling policies. But another way to solve this problem would be to make apt
more resilient about mirrors with heavy throttling policies. I can think of
- allow to specify a maximum bytes per second value for downloads (this has
the largest effect if set low enough)
That's Acquire::http::Dl-Limit
Post by Johannes Schauer
- allow to set an option that makes apt automatically retry when a transient
error occurs
That's Acquire::Retries - well it retries in general I suppose. It misses a bits
like DNS rotation, SRV rotation, but the goal is that this becomes the default
sometime next year, with 3 retries per URL or something to work with flaky mirrors.
Post by Johannes Schauer
- allow to set custom resolve addresses for domains like done in my code below
That we don't have. Gotta use /etc/hosts

Anyway, if you know you use snapshots.d.o, and it's flaky, maybe make use
of Dl-Limit and Retries option?
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Johannes Schauer
2020-09-14 21:50:01 UTC
Permalink
Quoting Julian Andres Klode (2020-09-14 21:49:15)
Post by Julian Andres Klode
Post by Johannes Schauer
- allow to specify a maximum bytes per second value for downloads (this
has the largest effect if set low enough)
That's Acquire::http::Dl-Limit
Post by Johannes Schauer
- allow to set an option that makes apt automatically retry when a transient
error occurs
That's Acquire::Retries - well it retries in general I suppose. It misses a bits
like DNS rotation, SRV rotation, but the goal is that this becomes the default
sometime next year, with 3 retries per URL or something to work with flaky mirrors.
Post by Johannes Schauer
- allow to set custom resolve addresses for domains like done in my code below
That we don't have. Gotta use /etc/hosts
Anyway, if you know you use snapshots.d.o, and it's flaky, maybe make use of
Dl-Limit and Retries option?
I stand corrected. I think apt already offers all the tools needed to use
snapshot.debian.org. Setting Acquire::http::Dl-Limit indeed does the job. And
the option is even documented in apt-transport-http(1)

Thanks!

cheers, josch

James Addison
2020-09-14 17:30:02 UTC
Permalink
Package: snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: ***@jp-hosting.net

Understood. The environment I'd like to get this working for is based on Debian stable, so we might be at an impasse unless I can compile the latest apt sources using buster. I'm making progress on that and attempting to resolve some time64-related compilation issues (unrelated to this bug).

I'd like to keep this issue open against the snapshot.debian.org meta package since that seems a good way to resolve the problem assuming it is purely server-side.

Given the discussion so far would it make sense if I use BTS to attempt to (re)assign this bug from packages {apt,snapshot.debian.org} -> {snapshot.debian.org}?
James Addison
2020-09-14 20:10:02 UTC
Permalink
Package: snapshot.debian.org
Followup-For: Bug #959518
X-Debbugs-Cc: ***@jp-hosting.net

(sorry: I realized I meant 'compile ... using bullseye' in that previous message, not buster (stable))
Loading...