Bug#1076449: mercurial: does not start anymore with python 3.12.4-3, hgdemandimport problem

[Daniel Serpell]

Package: mercurial
Version: 6.8-1
Severity: grave
Justification: renders package unusable

Dear maintainer,

In current sid, with python 3.12.4-3, mercurial fails at load with:

~$ hg
Traceback (most recent call last):
File "/usr/bin/hg", line 57, in <module>
from mercurial import dispatch
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module
super().exec_module(module)
File "<frozen importlib.util>", line 257, in exec_module
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module
super().exec_module(module)
File "<frozen importlib.util>", line 267, in exec_module
AttributeError: partially initialized module 'threading' has no attribute 'RLock' (most likely due to a circular import)
~$

Commenting out the hgdemandimport.enable() at line 55 of /usr/bin/hg, it works:

~$ hg
Mercurial Distributed SCM

basic commands:

add add the specified files on the next commit
annotate show changeset information by line for each file
clone make a copy of an existing repository
commit commit the specified files or all outstanding changes
diff diff repository (or selected files)
export dump the header and diffs for one or more changesets
forget forget the specified files on the next commit
init create a new repository in the given directory
log show revision history of entire repository or files
merge merge another revision into working directory
pull pull changes from the specified source
push push changes to the specified destination
remove remove the specified files on the next commit
serve start stand-alone webserver
status show changed files in the working directory
summary summarize working directory state
update update working directory (or switch revisions)

(use 'hg help' for the full list of commands or 'hg -v' for details)
~$

Tried this on my own machine and also in a newly installed VM.

Thanks,
Daniel.

-- System Information:
Debian Release: trixie/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.9.9-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mercurial depends on:
ii libc6 2.39-4
ii mercurial-common 6.8-1
ii python3 3.12.3-1
ii ucf 3.0043+nmu1

Versions of packages mercurial recommends:
ii openssh-client 1:9.7p1-7

Versions of packages mercurial suggests:
pn kdiff3 | kdiff3-qt | kompare | meld | tkcvs | mgdiff <none>
pn qct <none>

-- no debconf information

[Vincent Lefevre]

Post by Daniel Serpell
~$ hg
File "/usr/bin/hg", line 57, in <module>
from mercurial import dispatch
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module
super().exec_module(module)
File "<frozen importlib.util>", line 257, in exec_module
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "/usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py", line 52, in exec_module
super().exec_module(module)
File "<frozen importlib.util>", line 267, in exec_module
AttributeError: partially initialized module 'threading' has no attribute 'RLock' (most likely due to a circular import)
~$

Same problem with the previous version 6.7.4-1.

[Sam]

So, I have now two machines where this bug happens, but on a third one
(my notebook) there seems to be no problem, despite all of them showing
6.8-1 as the installed version (via `dpkg --list | grep mercurial`).

Any recommendations how to make a differential-diagnosis between two
systems? I think the only difference is that my notebook is a bit behind
in regards to updates through apt.

[Vincent Lefevre]

Post by Sam
So, I have now two machines where this bug happens, but on a third one
(my notebook) there seems to be no problem, despite all of them showing
6.8-1 as the installed version (via `dpkg --list | grep mercurial`).
Any recommendations how to make a differential-diagnosis between two
systems? I think the only difference is that my notebook is a bit behind
in regards to updates through apt.

Look at the version of the python3.12 packages. The issue occurs
with 3.12.4-3, but not with 3.12.4-1.

On my machines, downgrading all the packages of python3.12 source
from 3.12.4-3 to 3.12.4-1 made the issue disappear.

[Sam]

Follow up:
As a work-around, the following seems to get things to work again (not
tested in-depth though):


Change line /usr/lib/python3/dist-packages/hgdemandimport/demandimportpy3.py:32 from

_deactivated = True

to

_deactivated = False


This allows at least to hg command to be called, but I am not sure about
any further implications.

[Chris Markiewicz]

Post by Daniel Serpell
Package: mercurial
Version: 6.8-1
Severity: grave
Justification: renders package unusable
Dear maintainer,

Hi all,

I introduced the patches to Python to resolve race conditions in lazy
loading with threads. Initially, we imported threading at the module
level, but that broke gevent, so we moved the import inside the load.
It seems hgdemandimport turns all imports lazy, so if threading has
not already been imported, it will block its own import.

It looks like mercurial maintains a list of imports that cannot be
deferred, so the simplest thing is probably to update/patch that list
to include threading:

https://foss.heptapod.net/mercurial/mercurial-devel/-/blob/80c8dcf/hgdemandimport/__init__.py#L22-61

I'll see about whether we can also do something on the Python side to
consider this case, as I believe lazy-loader is looking into creating
similar functionality:
https://github.com/scientific-python/lazy-loader

Best,
Chris

[Vincent Lefevre]

Python 3.12.5 released with that patch, so I assume upstream will not fix
that.
The Debian bug contains a proposal for a less invasive fix in mercurial,
therefore re-assigning and keeping the bug open

Note that this has already been fixed in the mercurial package:

mercurial (6.8-2) unstable; urgency=high

* Fix demandimport breakage with recent python changes (see #1076449).

-- Julien Cristau <***@debian.org> Mon, 22 Jul 2024 18:26:26 +0200

Is there something else to do or should this bug just be closed?

[Matthias Klose]

Post by Vincent Lefevre

Python 3.12.5 released with that patch, so I assume upstream will not fix
that.
The Debian bug contains a proposal for a less invasive fix in mercurial,
therefore re-assigning and keeping the bug open

mercurial (6.8-2) unstable; urgency=high
* Fix demandimport breakage with recent python changes (see #1076449).
Is there something else to do or should this bug just be closed?

I thought, that the lazy loading was entirely disabled in mercurial,
that's why I kept this bug report open.