Discussion:
Bug#944242: fix build & test issues with Python 3.8
Add Reply
Matthias Klose
2019-11-06 15:50:02 UTC
Reply
Permalink
Package: src:python-biopython
Version: 1.73+dfsg-1
Severity: important
Tags: sid bullseye patch
User: debian-***@lists.debian.org
Usertags: python3.8

please find patches to build and run the tests with Python 3.8. All patches
taken from upstream except for
https://github.com/biopython/biopython/issues/2325
Andreas Tille
2019-11-19 08:40:03 UTC
Reply
Permalink
Hi,

it seems that BioPython 1.75 has solved all issues with Python 3.8.
However, when trying to build the Debian package I get the following
issues in the Build-Time test suite:

...
======================================================================
ERROR: Bio.KDTree._CKDTree
----------------------------------------------------------------------
Traceback (most recent call last):
File "run_tests.py", line 367, in runTest
optionflags=doctest.ELLIPSIS)
File "/usr/lib/python2.7/doctest.py", line 2412, in DocTestSuite
raise ValueError(module, "has no docstrings")
ValueError: (<module 'Bio.KDTree._CKDTree' from '/build/python-biopython-1.75+dfsg/.pybuild/cpython2_2.7/build/Bio/KDTree/_CKDTree.so'>, 'has no docstrings')
======================================================================
ERROR: Bio.Nexus.cnexus
----------------------------------------------------------------------
Traceback (most recent call last):
File "run_tests.py", line 367, in runTest
optionflags=doctest.ELLIPSIS)
File "/usr/lib/python2.7/doctest.py", line 2412, in DocTestSuite
raise ValueError(module, "has no docstrings")
ValueError: (<module 'Bio.Nexus.cnexus' from '/build/python-biopython-1.75+dfsg/.pybuild/cpython2_2.7/build/Bio/Nexus/cnexus.so'>, 'has no docstrings')
======================================================================
ERROR: Bio.PDB.QCPSuperimposer.qcprot
----------------------------------------------------------------------
Traceback (most recent call last):
File "run_tests.py", line 367, in runTest
optionflags=doctest.ELLIPSIS)
File "/usr/lib/python2.7/doctest.py", line 2412, in DocTestSuite
raise ValueError(module, "has no docstrings")
ValueError: (<module 'Bio.PDB.QCPSuperimposer.qcprotmodule' from '/build/python-biopython-1.75+dfsg/.pybuild/cpython2_2.7/build/Bio/PDB/QCPSuperimposer/qcprotmodule.so'>, 'has no docstrings')
======================================================================
ERROR: Bio.PDB.kdtrees
----------------------------------------------------------------------
Traceback (most recent call last):
File "run_tests.py", line 367, in runTest
optionflags=doctest.ELLIPSIS)
File "/usr/lib/python2.7/doctest.py", line 2412, in DocTestSuite
raise ValueError(module, "has no docstrings")
ValueError: (<module 'Bio.PDB.kdtrees' from '/build/python-biopython-1.75+dfsg/.pybuild/cpython2_2.7/build/Bio/PDB/kdtrees.so'>, 'has no docstrings')
----------------------------------------------------------------------
Ran 519 tests in 287.024 seconds

FAILED (failures = 4)
Skipping any tests requiring internet access
Python version: 2.7.17 (default, Oct 19 2019, 23:36:22)
[GCC 9.2.1 20191008]
...


Any hint how to solve this?

Greetings from Biohackathon

Andreas.
--
http://fam-tille.de
Peter Cock
2019-11-19 13:10:02 UTC
Reply
Permalink
I mean I would not worry about this particular test failing - and would
consider whitelisting this test acceptable.
https://github.com/peterjc/biopython/commit/5af680b5043c9f160a19e4bb0deab0ccc271280d
If that works, this pull request will apply the change upstream to Biopython:

https://github.com/biopython/biopython/pull/2347

Peter
Andreas Tille
2019-11-19 20:20:01 UTC
Reply
Permalink
Hi Peter,
I'd like to give you credit as the fastest upstream answering a
question. ;-) Thanks a lot for it!
Lucky timing.
Anyway: Thanks a lot.
Curious - do you have the Python 2.7 version and build details?
I've attached the full build log.
Thanks - I'v had a quick look. There must be something different to
how the build or installed directories are setup - our TravisCI tests
do not pick up the C modules at all.
Hmmm, I'd love to re-implement this for the Debian package.
Hmmm, you say its harmless but its breaking the build of the Debian
package anyway. So it would be good to have some means against it.
I mean I would not worry about this particular test failing - and would
consider whitelisting this test acceptable.
https://github.com/peterjc/biopython/commit/5af680b5043c9f160a19e4bb0deab0ccc271280d
Unfortunately this does not work. :-( I tried it in the packaging commit

https://salsa.debian.org/med-team/python-biopython/commit/bb94263daca0cd51968305805e444d0254c01c48
https://github.com/biopython/biopython/blob/biopython-175/Tests/run_tests.py#L151
Neither works this

https://salsa.debian.org/med-team/python-biopython/commit/e22d86592d4c29c723297d3d5eb9cc63aa6f8fb8
i.e. Add "Bio/KDTree/_CKDTree" etc to EXCLUDE_DOCTEST_MODULES
(with the proviso that to date I've only used this with Python modules,
you might need to include the .so extension?)
I'm not sure what you want to tell me with the last phrase in brackets.
Since you are asking about the 2.7 version: We need to get rid of
Python2 as soon as all reverse depends of biopython are ported to
Python3 (or removed from Debian). This might take some time but if
we could move this kind of doc string generation to be done by Python3
this would be some step in the right direction.
Do you have a list of things still depending on Biopython & Python 2.7
handy?
As far as I interpret

apt-cache rdepends python-biopython

it is

srst2
seqsero
prime-phylo
nanopolish

nothing that I would mind to crash for some time span. If it can not be
ported it will be removed anyway. So if you decide to drop Python2
support that would set a clear signal. If it is the safest means to
get rid of the above trouble I'm all for it.
We're discussing when exactly to drop Python 2.7 support -
with a final compatible release in December 2019 or January 2020
looking most likely.
Sounds sensible.

Kind regards

Andreas.
--
http://fam-tille.de
Andreas Tille
2019-11-19 22:30:01 UTC
Reply
Permalink
Do you have a list of things still depending on Biopython & Python 2.7
handy? We're discussing when exactly to drop Python 2.7 support -
with a final compatible release in December 2019 or January 2020
looking most likely.
Speaking about droping Python2 and just running tests with Python3
I was running into other errors:

======================================================================
FAIL: kmedoids (Bio.Cluster._cluster)
Doctest: Bio.Cluster._cluster.kmedoids
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 2196, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for Bio.Cluster._cluster.kmedoids
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line unknown line number, in kmedoids

----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.kmedoids
Failed example:
distance = array([[0.0, 1.1, 2.3],
[1.1, 0.0, 4.5],
[2.3, 4.5, 0.0]])
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.kmedoids[0]>", line 1, in <module>
distance = array([[0.0, 1.1, 2.3],
NameError: name 'array' is not defined
----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.kmedoids
Failed example:
distance = array([1.1, 2.3, 4.5])
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.kmedoids[1]>", line 1, in <module>
distance = array([1.1, 2.3, 4.5])
NameError: name 'array' is not defined
----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.kmedoids
Failed example:
distance = [array([]),
array([1.1]),
array([2.3, 4.5])]
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.kmedoids[2]>", line 1, in <module>
distance = [array([]),
NameError: name 'array' is not defined


======================================================================
FAIL: pca (Bio.Cluster._cluster)
Doctest: Bio.Cluster._cluster.pca
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 2196, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for Bio.Cluster._cluster.pca
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line unknown line number, in pca

----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.pca
Failed example:
columnmean + dot(coordinates, pc)
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.pca[0]>", line 1, in <module>
columnmean + dot(coordinates, pc)
NameError: name 'columnmean' is not defined


======================================================================
FAIL: treecluster (Bio.Cluster._cluster)
Doctest: Bio.Cluster._cluster.treecluster
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 2196, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for Bio.Cluster._cluster.treecluster
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line unknown line number, in treecluster

----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.treecluster
Failed example:
distance = array([[0.0, 1.1, 2.3],
[1.1, 0.0, 4.5],
[2.3, 4.5, 0.0]])
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.treecluster[0]>", line 1, in <module>
distance = array([[0.0, 1.1, 2.3],
NameError: name 'array' is not defined
----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.treecluster
Failed example:
distance = array([1.1, 2.3, 4.5])
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.treecluster[1]>", line 1, in <module>
distance = array([1.1, 2.3, 4.5])
NameError: name 'array' is not defined
----------------------------------------------------------------------
File "/build/python-biopython-1.75+dfsg/.pybuild/cpython3_3.8/build/Bio/Cluster/_cluster.cpython-38-x86_64-linux-gnu.so", line ?, in Bio.Cluster._cluster.treecluster
Failed example:
distance = [array([]),
array([1.1]),
array([2.3, 4.5])]
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.8/doctest.py", line 1328, in __run
exec(compile(example.source, filename, "single",
File "<doctest Bio.Cluster._cluster.treecluster[2]>", line 1, in <module>
distance = [array([]),
NameError: name 'array' is not defined


----------------------------------------------------------------------
Ran 521 tests in 210.480 seconds

FAILED (failures = 1)


Kind regards

Andreas.
--
http://fam-tille.de
Andreas Tille
2019-11-20 12:40:02 UTC
Reply
Permalink
Control: forwarded -1 https://github.com/biopython/biopython/issues/2350
Andreas Tille
2019-12-02 15:40:01 UTC
Reply
Permalink
Hi Peter,
Post by Andreas Tille
https://github.com/biopython/biopython/issues/2350
Since I urgently need Biopython >= 1.74 to fix some other bugs I decided
to try to ignore those issues we discussed here and made sure that build
time tests will not fail. However the Debian package is also running a
Continuous Integration test (autopkgtest in Debian terminology). The
test for biopython is basically copying the test files to a temporary dir
and execute the tests[1].

In the attached log I injected a `set -x` and was running:

sh run-unit-test 2>&1 | tee > tes_suite_errors.out

Unfortunately there are more errors than I expected. Do you have any
idea why?

Kind regards

Andreas.


[1] https://salsa.debian.org/med-team/python-biopython/blob/master/debian/tests/run-unit-test
--
http://fam-tille.de
Andreas Tille
2019-12-09 07:40:01 UTC
Reply
Permalink
Control: forwarded -1 https://github.com/biopython/biopython/issues/2388

There is an autopkgtest regression on arm64 that prevents migration:

https://ci.debian.net/data/autopkgtest/testing/arm64/p/python-biopython/3607200/log.gz

It is reported upstream as issue 2388
--
http://fam-tille.de
Loading...