Discussion:
Bug#930487: lintian: speed up test suite CI
Add Reply
Dmitry Bogatov
2019-06-13 16:00:01 UTC
Reply
Permalink
Package: lintian
Version: 2.15.0
Severity: wishlist

Dear Maintainer,

Gitlab CI jobs take very long to complete: around 1.5 hours. No wonder
-- test suite builds generates and builds more then 900 source packages,
and after that it runs Lintian in those source packages.

As can be seen here[1], build of source packages takes quite significant
portion of total run time: around 33 minutes. It would be great to use
Gitlab caching to avoid rebuiling package at every job run.

[1] https://salsa.debian.org/kaction/lintian/-/jobs/195835
Felix Lechner
2019-06-13 18:50:01 UTC
Reply
Permalink
It would be great to ... avoid rebuiling package at every job run.
I would be nice to see how other projects deal with this issue. There
is some support for uploading the test packages separately. [1]

For Lintian, however, I would prefer to upload the test packages
separately to Debian's regular build infrastructure. The test packages
can and do have conflicting build dependencies and architectures.
These conflicts are not addressed currently, and may require separate
chroot build environments. It would be difficult to implement even in
a single bulk package. Debian's infrastructure, on the other hand, is
designed to build the packages.

At the same time, separate uploads would place an undue burden on the
archive's namespace and on the NEW queue. There would also be delays
for new tags, as Lintian may at some point require that tags are
tested. All test packages would have to be in the archive before the
lintian source is uploaded.

Right now, my favorite solution would be for the archive to offer
dependent namespaces for source packages (such as lintian/...). Such
internal packages could be uploaded separately and would not have to
go through the NEW queue. Outside packages could not depend on them,
but they would be installed if their source package requires them.

This idea will likely generate much opposition. Let me just say that I
am not sure my suggestion is worth the effort, or useful for anyone
else.

Kind regards,
Felix

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926409#42
Chris Lamb
2019-06-13 20:40:01 UTC
Reply
Permalink
Post by Felix Lechner
For Lintian, however, I would prefer to upload the test packages
separately to Debian's regular build infrastructure.
Hm, I think you are talking at cross-purposes to Dmitry here. Nobody
was suggesting we upload the test packages to Debian; that would
surely be impossible.

Gitlab has a support for saving various parts of a successful build
for the next one. I believe the idea is that we would build the test
packages and then push them to this cache re-using them on any subsequent
test runs. People often use this to cache "pip" Python dependencies
but I don't see any obvious reason why we can't use it here.

(This is indexed by some cache key so that changing the testsuite
itself could/would rebuild the packages as appropriate.)


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Dmitry Bogatov
2019-06-17 00:50:01 UTC
Reply
Permalink
Post by Chris Lamb
Post by Felix Lechner
For Lintian, however, I would prefer to upload the test packages
separately to Debian's regular build infrastructure.
Hm, I think you are talking at cross-purposes to Dmitry here. Nobody
was suggesting we upload the test packages to Debian; that would
surely be impossible.
Gitlab has a support for saving various parts of a successful build
for the next one. I believe the idea is that we would build the test
packages and then push them to this cache re-using them on any subsequent
test runs. People often use this to cache "pip" Python dependencies
but I don't see any obvious reason why we can't use it here.
Thank you. That is exactly what I meant.
--
Note, that I send and fetch email in batch, once in a few days.
Chris Lamb
2019-06-17 23:10:02 UTC
Reply
Permalink
Post by Chris Lamb
Gitlab has a support for saving various parts of a successful build
for the next one. I believe the idea is that we would build the test
packages and then push them to this cache re-using them on any subsequent
test runs.
[..]

The naïve solution here might be to save & restore debian/test-out
between runs. However, I'm not sure whether this would result in
changes to the testsuite itself resulting in the changed versions
being tested, resulting in all manner of false positive and false
negatives.

Felix, any insight here? There's currently no magical command in the
fancy test runner of yours that will rebuild any missing or otherwise
changed test packages is there…?

Or, before we implement that, am I asking an XY question? In other
words, are we barking up the wrong tree here and what we need to do is
use different GitLab CI stage altogether and pass "artifacts" around
instead?

https://docs.gitlab.com/ee/ci/caching/index.html#cache-vs-artifacts

I'm not entirely sure, alas. (Although that magical command might be
useful locally...)


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Felix Lechner
2019-06-18 00:10:01 UTC
Reply
Permalink
Post by Chris Lamb
The naïve solution here might be to save & restore debian/test-out
between runs.
I do not think that will work for very long, although there is a
better chance if we separate the expected tags from the artifact
directory. (The expected tags change relatively often a new checks are
implemented or old ones are tweaked.)
Post by Chris Lamb
There's currently no magical command in the
fancy test runner of yours that will rebuild any missing or otherwise
changed test packages is there…?
We used filesystem timestamps for a while, but the standard resolution
(1 sec) was not granular enough. AFAIR, we now generate everything
every time. We instead split the generation of test packages from the
test runs, although they currently just run consecutively. We could
probably skip the generation of test packages if they are already
present and nothing in t/ has changed.
Post by Chris Lamb
In other
words, are we barking up the wrong tree here and what we need to do is
use different GitLab CI stage altogether and pass "artifacts" around
instead?
https://docs.gitlab.com/ee/ci/caching/index.html#cache-vs-artifacts
Artifacts may work, but uploading them separately without a dependency
scheme seems to invite other problems. Also, your local build
architecture and environment—which may figure into the artifacts you
upload—may not match what the the runner needs. (I am thinking about
stable or ubuntu-devel.)
Chris Lamb
2019-06-18 09:10:01 UTC
Reply
Permalink
[changing subject to match updated bug title]

Hi Felix,
Post by Felix Lechner
We used filesystem timestamps for a while, but the standard resolution
(1 sec) was not granular enough. AFAIR, we now generate everything
every time.
By "filesystem timestamps" here are you referring to comparing the
timestamp of each generated test package and its source? If so, I am
unclear why one second was not enough to determine whether a *developer*
had changed something under "t/".
Post by Felix Lechner
We could probably skip the generation of test packages if they are
already present and nothing in t/ has changed.
Indeed, but that is (roughly) what this entire bug report is about.
Post by Felix Lechner
But are we barking up the wrong tree here and what we need to do is
use different GitLab CI stage altogether and pass "artifacts" around
instead?
https://docs.gitlab.com/ee/ci/caching/index.html#cache-vs-artifacts
Artifacts may work, but uploading them separately without a dependency
scheme seems to invite other problems. […]
Hm, I think you may have been accidentally misled about how the Gitlab
CI build stages — there would be no upload whatsoever. I am therefore
unsure how to effectively and productively respond to your remarks,
alas. :(


Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Dmitry Bogatov
2019-07-04 17:50:01 UTC
Reply
Permalink
Post by Chris Lamb
Gitlab has a support for saving various parts of a successful build
for the next one. I believe the idea is that we would build the test
packages and then push them to this cache re-using them on any subsequent
test runs. People often use this to cache "pip" Python dependencies
but I don't see any obvious reason why we can't use it here.
What is Lintian policy on usage of languages other then Perl?
It feels that build system for test suite is overconstrained now
(rebuilds more then necessary), and I consider to make some experiments
with Shake[1]. Should the experiments succeed, will they be accepted?

[1] https://shakebuild.com
--
Note, that I send and fetch email in batch, once in a few days.
Please, mention in body of your reply when you add or remove recepients.
Felix Lechner
2019-07-04 18:50:01 UTC
Reply
Permalink
Hi Dmitry,
Post by Dmitry Bogatov
It feels that build system for test suite is overconstrained now
(rebuilds more then necessary), and I consider to make some experiments
with Shake[1]. Should the experiments succeed, will they be accepted?
The large majority of test packages is built using
'dpkg-buildpackage'. Do you plan to rewrite it?

I looked into using golang for the test suite but, for the things we
are doing, Perl is quite fast. It's also mature and available
everywhere. How does Shake compare?

Kind regards,
Felix
Chris Lamb
2019-07-04 19:40:01 UTC
Reply
Permalink
Hi Dmitry,
Post by Dmitry Bogatov
What is Lintian policy on usage of languages other then Perl?
I'm very much in favour of us exhausting all caching opportunies, both
on our CI system and locally, before we introduce another language and
all the complications that would entail.


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Felix Lechner
2019-07-04 19:50:01 UTC
Reply
Permalink
Hi Chris,
Post by Chris Lamb
I'm very much in favour of us exhausting all caching opportunies, both
on our CI system and locally,
Would it help to separate the build product for each test package,
which will presumably be cached, from the test output (tags.actual,
tagdiff and friends)?

Kind regards
Felix
Chris Lamb
2019-07-04 21:10:01 UTC
Reply
Permalink
Hi Felix,
Post by Felix Lechner
Post by Chris Lamb
I'm very much in favour of us exhausting all caching opportunies, both
on our CI system and locally,
Would it help to separate the build product for each test package,
which will presumably be cached, from the test output (tags.actual,
tagdiff and friends)?
Let's zoom out here so we are not asking "Xy questions" of each other.

I think the most common use-case is that I make some change — possibly
extremely minor — in, say, checks/foo.pm and I want to re-run the
testsuite to check that I've fixed whatever false-positive or edge-case
in a tag that I am in the process of implementing. We should optimise
for kind of pattern. What do you think?

What this means in terms of the implementation detail of the testsuite
(which I'm afraid I have let get beyond my insight) I would have to
leave up to you, alas.


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Dmitry Bogatov
2019-07-07 06:40:02 UTC
Reply
Permalink
@Felix

I think, that your proposal of caching test binary package separately
from test output would be improvement. Maybe it would even be "good
enough".

I will take a look (no promises made) into this direction. But
ideally...
Post by Chris Lamb
I think the most common use-case is that I make some change — possibly
extremely minor — in, say, checks/foo.pm and I want to re-run the
testsuite to check that I've fixed whatever false-positive or edge-case
in a tag that I am in the process of implementing. We should optimise
for kind of pattern. What do you think?
Ideally, I want test suite automatically detect:

* new test directory created, test.deb needs to be built
* test directory edited, test.deb needs to be rebuilt
* this configuration was already checked, nothing to do

+ I do not want re-run perl-critic and perltidy checks aganist files,
that were not changed
+ ...

In short, I want test suite to be dependable build system from source
files and test files into tarball of test logs. (~950 files or so)

Right now it feels like build system written in Make -- undercontrained
in one cases (you can easily get stalled results), overconstrained in
another cases (you build more then necessary -- perlcritic tests, for
example).

To get feeling what is perfect (in sense of dependencies correctness)
build system, take a look at `tup'[1]. Read the tutorial, it is worthy
reading.

Unfortunately for our case, `tup' only supports static dependencies and
outputs (applicative), so you can't do following with it:

* foo.txt contains list of files
* foo.tar contains files, listed in foo.txt
* foo.tar perfectly depends on foo.txt and files, listed in it.

That is why I talk about Shake -- it is more complicated, compared to
tup, but it supports dynamic (monadic) rules, so previous example with
`foo.tar' is basic example of Shake.

In theory, it is possible to implement all features I mentioned with
ad-hoc Perl code, but I wouldn't volonteer to do so.

@Felix:

As to question of maturity, Shake is Haskell library, implemented and
maintainer by well-known and well-esteemed developer. I do not expect
it disappear anytime soon.

Of course, I do not plan (or propose) to re-implement `dpkg-buildpackage',
I just want it to be called when and only when it is necessary.

[1] http://gittup.org/tup
--
Note, that I send and fetch email in batch, once in a few days.
Please, mention in body of your reply when you add or remove recepients.
Felix Lechner
2019-07-07 12:10:02 UTC
Reply
Permalink
Hi Dmitry,
Post by Dmitry Bogatov
Post by Chris Lamb
I think the most common use-case is that I make some change — possibly
extremely minor — in, say, checks/foo.pm and I want to re-run the
testsuite to check that I've fixed whatever false-positive or edge-case
in a tag that I am in the process of implementing. We should optimise
for kind of pattern. What do you think?
In that scenario, I usually restrict the run with
"--onlyrun=check:manpages" (after editing checks/manpages.pm). You can
also use the selectors 'test:' or 'tag:'.
Post by Dmitry Bogatov
* test directory edited, test.deb needs to be rebuilt
+ I do not want re-run perl-critic and perltidy checks aganist files,
that were not changed
I have not found a way to implement that. An incremental builder I
wrote based on file modification times (in
lib/Test/StagedFileProducer.pm) cannot tell the difference between two
files that were built within one second of another. That is the
granularity of Linux filesystems. The templating mechanism relies on
many intermediate build products. There should be a way to rebuild a
test case if any of the inputs have changed, but I haven't found it.
Patches are welcome.
Post by Dmitry Bogatov
In short, I want test suite to be dependable build system from source
files and test files into tarball of test logs. (~950 files or so)
I think it's more or less dependable. Our issue is speed.

You can find logs for each test case in ./debian/test-out/.../log.
Post by Dmitry Bogatov
Right now it feels like build system written in Make -- undercontrained
in one cases (you can easily get stalled results), overconstrained in
another cases (you build more then necessary -- perlcritic tests, for
example).
I think the test suite builds exactly what is necessary. It just takes
a long time. :)

Perlcritic (or more often perltidy) is something we impose on
ourselves. There are maddening moments, particularly when the software
corrects itself, but on balance I am in favor.
Post by Dmitry Bogatov
To get feeling what is perfect (in sense of dependencies correctness)
build system, take a look at `tup'[1]. Read the tutorial, it is worthy
reading.
How does 'tup' deal with the timestamp granularity? Is that solved
with a 'directed acyclic graph (DAG)'?
Post by Dmitry Bogatov
That is why I talk about Shake -- it is more complicated, compared to
tup, but it supports dynamic (monadic) rules, so previous example with
`foo.tar' is basic example of Shake.
Perl is a generalized programming language (and Debian runs on it).
Everyone can contribute. I think the best path forward is to try
caching, as Chris suggested. If that does not work, we should consider
building the test packages separately.

Thank you for your interest in the Lintian test suite. The long build
times bother many people. I would very much like to reduce them.
Chris Lamb
2019-07-20 20:40:01 UTC
Reply
Permalink
Hi all,
lintian: use GitLab caching of test packages to speed up test suite CI
I've been hacking on this on a Salsa-local fork of Lintian that splits
the generation of the test packages and the testing itself, crucially
caching the result of the former if the tests and some other key files
have not changed.

I have something working except that I am running into a blocker
whereby the GitLab cache is not seen in subsequent runs.

For example, the cache is being stored at the end of the test package
build stage:

Creating cache default-2...
.cache/: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache

… but it is not seen when you re-run that same pipeline:

Checking cache for default-2...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache

$ ls -l .cache || true
ls: cannot access '.cache': No such file or directory

I will try and hunt down a Salsa admin during DebConf. This might be
due to the use of shared runners and us not using a centralised cache.


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org / chris-lamb.co.uk
`-
Felix Lechner
2019-07-20 21:10:02 UTC
Reply
Permalink
Hi Chris,
Post by Chris Lamb
I've been hacking on this on a Salsa-local fork of Lintian that splits
the generation of the test packages and the testing itself, crucially
caching the result of the former if the tests and some other key files
have not changed.
How do you tell when files have changed? I would like to use an MD5
manifest of all files generated after templating. That will also catch
changes in defaults and templates---even missing files.

Eventually, the expected tags (or output, if applicable) should be
excluded from that manifest, as should those portions of each 'desc'
file that apply only to lintian invocation (i.e. Profile,
Output-Format) or runner behavior (i.e. Match-Strategy,
Test-Architectures). This will require a splitting of all 'desc' files
in the test directories. Also, the specifications for test packages
should then be moved into dedicated directories in each test, named
'$test/test-object' or similar.

Kind regards,
Felix
Felix Lechner
2019-07-20 21:20:02 UTC
Reply
Permalink
On Sat, Jul 20, 2019 at 2:02 PM Felix Lechner
Post by Felix Lechner
This will require a splitting of all 'desc' files
in the test directories.
Actually, since the values from 'desc' are incorporated in the
completed templates, all 'desc' files can be excluded from the
manifest as well. No splitting is necessary.

Kind regards,
Felix
Chris Lamb
2019-07-21 13:50:01 UTC
Reply
Permalink
Post by Felix Lechner
Post by Chris Lamb
I've been hacking on this on a Salsa-local fork of Lintian that splits
the generation of the test packages and the testing itself, crucially
caching the result of the former if the tests and some other key files
have not changed.
How do you tell when files have changed? I would like to use an MD5
manifest of all files generated after templating.
Computing the checksum of the output not only seems unnecessarily
complicated but prone to error in the details. I am hashing the entire
"input" roughly as follows:

$ find t/ debian/ -type f | sort | xargs sha1sum | sha1sum | cut -d' ' -f1

.. throwing the entire set of build packages away in the case of cache
miss.

Of course there are more refined approaches but, in lieu of that, if
you skim over the pipeline job history this will save the testsuite
package builds almost all of the time. This also has the advantage of
being obvious how it works and captures any type of change, including
the missing files that you specifically highlight, but possibly even
others we have not yet thought of.

I am very much disposed towards simple and braindead obvious in things
like this, especially when they hit 90% of the use-case. And then we
can get back to other stuff...

Anyway, without the Salsa caching working, all of this is moot.


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Felix Lechner
2019-07-21 14:20:01 UTC
Reply
Permalink
Hi,
Post by Chris Lamb
$ find t/ debian/ -type f | sort | xargs sha1sum | sha1sum | cut -d' ' -f1
That's the same approach I suggested, except I would save the result of

find t/ debian/ -type f | sort | xargs sha1sum

and do so on a per-test basis. Then t/bin/runtest can rebuild the test
packages adaptively.

Please remember, contributors currently cannot run tests with
'--onlyrun=check:XXX' when they add or change a tag, except after the
time-consuming 'rm -rf debian/test-out; t/bin/build-test-packages'.
(It was much faster until I separated the building of test packages.)
The adaptive rebuilding would be super helpful even without caching on
Salsa.
Post by Chris Lamb
.. throwing the entire set of build packages away in the case of cache
miss.
I did not suggest by-package caching on Salsa. I would write the test
packages in to a different directory tree in d/test-out and only cache
that.

For example, I would write the filled templates to d/test-out/src.
Only this tree needs to be hashed. During building, I would then write
the test packages to d/test-out/built. Only that tree needs to be
cached.
Post by Chris Lamb
Anyway, without the Salsa caching working, all of this is moot.
One small step for a man; one big step for mankind. :)
Felix Lechner
2019-07-21 16:10:01 UTC
Reply
Permalink
Hi
Post by Chris Lamb
I am very much disposed towards simple and braindead obvious in things
like this, especially when they hit 90% of the use-case. And then we
can get back to other stuff...
I realized we are talking about related but different things. Please
look out for a merge request on the topics that go beyond the scope of
this bug.
Georg Faerber
2019-07-24 22:00:02 UTC
Reply
Permalink
Hi,

I'm not sure if the following is of help, as I didn't saw the involved
Post by Chris Lamb
I have something working except that I am running into a blocker
whereby the GitLab cache is not seen in subsequent runs.
For example, the cache is being stored at the end of the test package
Creating cache default-2...
.cache/: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Checking cache for default-2...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ ls -l .cache || true
ls: cannot access '.cache': No such file or directory
I will try and hunt down a Salsa admin during DebConf. This might be
due to the use of shared runners and us not using a centralised cache.
An alternative might be to use artifacts [1], which work as expected on
Salsa.

Cheers,
Georg


[1] https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html
Chris Lamb
2019-07-25 12:20:01 UTC
Reply
Permalink
Hi Georg,
Post by Georg Faerber
An alternative might be to use artifacts [1], which work as expected on
Salsa.
Thanks. Unfortunately, I did investigate using artifacts at first and
whilst they do indeed work on Salsa they are for a different use-case
that does not fit our "re-use from a previous build" requirement (they
re-use files between *parts* of a build).


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Felix Lechner
2019-07-30 13:20:01 UTC
Reply
Permalink
Hi Chris,
Post by Chris Lamb
I've been hacking on this on a Salsa-local fork of Lintian that splits
the generation of the test packages and the testing itself, crucially
caching the result of the former if the tests and some other key files
have not changed.
Will it also force an update of the test packages when the build chain
has changed? It may expose Lintian bugs like #932339.

Kind regards,
Felix
Chris Lamb
2019-07-31 00:50:01 UTC
Reply
Permalink
Post by Felix Lechner
Post by Chris Lamb
caching the result of the former if the tests and some other key files
have not changed.
Will it also force an update of the test packages when the build chain
has changed? It may expose Lintian bugs like #932339.
Whilst my *current* implementation does not rebuild if a build-depends
version changes, we would only need to include these package version
in the hash key. It would then (correctly) cache miss in that instance
and rebuild the test packages. It's a shame we would not have
a .buildinfo file at this point...

I'm not going to implement this bit just yet until I solve the more
deeper problem of it, well, not actually caching anything at the
moment. See previous messages for more background.


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org 🍥 chris-lamb.co.uk
`-
Georg Faerber
2019-07-31 12:40:02 UTC
Reply
Permalink
Hi again,
Post by Chris Lamb
I'm not going to implement this bit just yet until I solve the more
deeper problem of it, well, not actually caching anything at the
moment. See previous messages for more background.
As far as I can tell, the cache problem was caused by a breaking
upstream change. This got fixed [1]. Probably, didn't test, this "should
work" again.

Cheers,
Georg


[1] https://salsa.debian.org/salsa/salsa-ansible/commit/c6404e6b31267a8f9c46a2eb7838b829e08a451d
Loading...