Bug#1080168: debmirror: Debmirror does not download or generate contents files for bookworm-proposed-updates, even with --getcontents

[charles]

Package: debmirror
Version: 1:2.41
Severity: normal
X-Debbugs-Cc: ***@gmail.com

Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

* What led up to the situation?
Added proposed updates to my mirror setup, which completed with out errors.
However apt complains about missing contents files for bookworm-proposed-
updates.

* What exactly did you do (or not do) that was effective (or
ineffective)?
Attempted various combinations of diff options and --getcontents.

* What was the outcome of this action?
Debmirror appears to download the needed files for all the sections other than
proposed-updates

* What outcome did you expect instead?
Hoping I have something configured incorrectly here, but I have been unable to
find out what in my testing.

*** End of the template - remove these template lines ***


-- System Information:
Debian Release: 12.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'proposed-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.10.6+bpo-amd64 (SMP w/32 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_WARN, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages debmirror depends on:
ii bzip2 1.0.8-5+b1
pn libdigest-md5-perl <none>
pn libdigest-sha-perl <none>
ii liblockfile-simple-perl 0.208-1.1
ii libwww-perl 6.68-1
ii perl [libnet-perl] 5.36.0-7+deb12u1
ii rsync 3.2.7-1
ii xz-utils 5.4.1-0.2

Versions of packages debmirror recommends:
ii ed 1.19-1
ii gpgv 2.2.40-1.1
ii patch 2.7.6-7

Versions of packages debmirror suggests:
ii gnupg 2.2.40-1.1

-- no debconf information

[Colin Watson]

Post by charles
Added proposed updates to my mirror setup, which completed with out errors.
However apt complains about missing contents files for bookworm-proposed-
updates.

Could you please send to this bug your debmirror configuration file (if
any) and the full debmirror command line you're using?

Thanks,

[Charles Laws]

Apologies for the missing information,

Debmirror is being called by a bash script with:

#!/bin/bash
#
debmirror --keyring /usr/share/keyrings/debian-archive-keyring.gpg
--config-file /home/debmirror/debmirror-debian.conf

Contents of config file:

# Default config for debmirror

# The config file is a perl script so take care to follow perl syntax.
# Any setting in /etc/debmirror.conf overrides these defaults and
# ~/.debmirror.conf overrides those again. Take only what you need.
#
# The syntax is the same as on the command line and variable names
# loosely match option names. If you don't recognize something here
# then just stick to the command line.
#
# Options specified on the command line override settings in the config
# files.
# Location of the local mirror (use with care)
$mirrordir="/home/debmirror/mirrors/debian/";

# Output options
#$verbose=0;
#$progress=0;
#$debug=0;

# Download options
$host="mirrors.lug.mtu.edu";
#$user="anonymous";
#$passwd="anonymous@";
$remoteroot="debian";
$download_method="rsync";
@dists="bookworm,bookworm-backports,bookworm-updates,bookworm-proposed-updates";
@sections="main,main/debian-installer,contrib,non-free,non-free-firmware";
@arches="i386,amd64,arm64";
# @ignores="";
# @excludes="";
# @includes="";
# @excludes_deb_section="";
# @limit_priority="";
#$omit_suite_symlinks=0;
#$skippackages=0;
@rsync_extra="doc,tools,trace";
#$i18n=0;
$getcontents=1;
$do_source=0;
#$max_batch=0;

# @di_dists="dists";
# @di_archs="arches";

# Save mirror state between runs; value sets validity of cache in days
$state_cache_days=0;

# Security/Sanity options
#$ignore_release_gpg=0;
#$ignore_release=0;
#$check_md5sums=0;
#$ignore_small_errors=0;

# Cleanup
#$cleanup=0;
#$post_cleanup=1;

# Locking options
#$timeout=300;

# Rsync options
#$rsync_batch=200;
#$rsync_options="-aIL --partial";
$rsync_options="-aIL --partial --bwlimit=5000 --no-motd";

# FTP/HTTP options
#$passive=0;
# $proxy="http://proxy:port/";

# Dry run
#$dry_run=0;

# Don't keep diff files but use them
#$diff_mode="use";
$diff_mode="mirror";

# The config file must return true or perl complains.
# Always copy this.
1;

Regards,

Charles

[Charles Laws]

Package: debmirror
Tags: patch

[Colin Watson]

Admittedly I am a novice at coding, but I believe I have found the issue.
Adjusting the following appears to allow debmirror to download the
contents files from proposed-updates.
--- debmirror 2024-09-09 13:13:06.335485914 -0600
+++ debmirror.revised 2024-09-09 13:15:21.742485255 -0600
@@ -3301,7 +3301,7 @@
   foreach my $dist (keys %distset) {
-    next if $dist=~/\bexperimental\b|-proposed-updates\b/o;
+    next if $dist=~/\bexperimental\bproposed-updates\b/o;

You're right that this is the cause of the problem, but this isn't quite
the right fix: it excludes --dist items that match a regular expression
defined as "experimental with word boundaries around it, immediately
followed by proposed-updates with word boundaries around it" - it's
literally impossible for this regular expression ever to match any
string, so this can't be what we should do.

When faced with this sort of problem, it's usually best to look back
through history to find out what the offending code was doing in the
first place. Unfortunately doing that exercise here leads us back to a
period where our version control history wasn't very detailed (because
the history was imported from a time before debmirror was
version-controlled at all, I think), but the commit in question is:

https://salsa.debian.org/debian/debmirror/-/commit/10a29dcd2995d6b8a2def028eb46dc49d442d08e

And looking in debian/changelog, there's this entry under 20040926:

* Skip Contents files for *-proposed-updates and experimental

So I think the chances are that -proposed-updates didn't have Contents
files back then. https://archive.debian.org/debian/dists/ doesn't have
complete records here, but they appear to have been added some time
between squeeze (Debian 6.0) and stretch (Debian 9).

However, nowadays debmirror skips files if they don't exist, without
having to explicitly exclude them from consideration up-front. So the
right fix is just to remove this line entirely, and I'll do that.

Thanks for the investigation!