Discussion:
Bug#868527: want sbuild --no-source or something
(too old to reply)
Ian Jackson
2017-07-16 12:40:01 UTC
Permalink
Raw Message
Package: sbuild
Version: 0.73.0-4
Severity: wishlist

It would be nice if it were easier to use sbuild with a gitish
downstream workflow which does not produce "3.0 (quilt)" source
packages.

For example, consider this scenario:

dgit clone foo stretch
cd foo
hack hack hack
git commit
git merge upstream/stable
hack hack hack
git commit
gbp dch
sbuild -A
dpkg -iGOEB ../*.deb

This is similar to the recommendatioms in dgit-user(7), except that we
are trying to use sbuild.

The above attempt will probably fail.

This is because the package "foo" is probably "3.0 (quilt)", according
to debian/source/format. sbuild will try to build a source package,
and dpkg-source will want the user's changes to be "committed" as
patches in debian/patches. The user won't have done that and probably
doesn't want to. Worse, in the general case, changes might be made to
the git tree which dpkg-source cannot represent.

The solution is to not build a source package. The user who adopts a
gitish workflow probably doesn't want one. However, sbuild needs a
source package because that's how it transfers the package into the
chroot.

I suggest that sbuild grow an option which generates a dummy source
package in some hidden or temporary place. I think the actual source
package generation can be done with something like this rune (and that
this will always succeed):
dpkg-source -Zgzip -z0 --format=1.0 -sn
(but I have not yet tested this myself).

I don't have much of an opinion what the sbuild option should be
called. --no-source is a possibility. Or maybe it should be inferred
from the other options and the fact that sbuild is working from a
directory tree - although that would be an incompatible change.

In the meantime, I think users can use
sbuild -A --dpkg-source-opts='-Zgzip -z0 --format=1.0 -sn'
However, that sets up a hazard: it produces a dummy 1.0 native tarball
package as if it were a build product. But that native tarball
package is broken: it can be extracted and built, but it cannot be
re-packed, without again providing special optionsn to dpkg-source.

Thanks,
Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Johannes Schauer
2017-07-16 20:30:01 UTC
Permalink
Raw Message
Quoting Ian Jackson (2017-07-16 14:32:17)
Post by Ian Jackson
It would be nice if it were easier to use sbuild with a gitish
downstream workflow which does not produce "3.0 (quilt)" source
packages.
[snip]
The above attempt will probably fail.
This is because the package "foo" is probably "3.0 (quilt)", according
to debian/source/format. sbuild will try to build a source package,
and dpkg-source will want the user's changes to be "committed" as
patches in debian/patches. The user won't have done that and probably
doesn't want to. Worse, in the general case, changes might be made to
the git tree which dpkg-source cannot represent.
The solution is to not build a source package. The user who adopts a
gitish workflow probably doesn't want one. However, sbuild needs a
source package because that's how it transfers the package into the
chroot.
Indeed, the task to solve is how to transfer the source into the chroot.

But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?
Would it not be more fitting to use some git-based command to exchange the
data? For example using a git bundle? This would then make sure that everything
git knows about is transferred into the chroot. Sbuild can make sure that git
gets installed together with build-essential and is thus able to git unbundle
the file.

What do you think?

Thanks!

cheers, josch
Raphael Hertzog
2017-07-17 08:30:01 UTC
Permalink
Raw Message
Hi,
Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?
No, it's clearly a hack and one that might have side effects due to
differences (for example in set of ignored files) between format 1.0
and format 3.0 (quilt).

To me the right approach is to extend sbuild to support alternative ways
to copy the unpacked source tree into the build chroot. Your suggestion
to rely on a git bundle is a possibility. A plain tar of the build
directory is another approach.

IMO this goes in the right direction of entirely eliminating source
packages in the long term.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/
Ian Jackson
2017-07-17 12:40:02 UTC
Permalink
Raw Message
Post by Raphael Hertzog
Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?
No, it's clearly a hack and one that might have side effects due to
differences (for example in set of ignored files) between format 1.0
and format 3.0 (quilt).
Indeed. I think it's the best recommendation, though, with the
current tooling. Unless you can think of a better option, I think I
should recommend this hack to stretch users.

You are right that it has differences in the ignored files.
Specifically, this rune transfers the complete contents of the working
tree, including the .git directory, into the build environment. I
doubt that is likely to cause trouble in practice.

Ian.

Ian Jackson
2017-07-17 09:40:01 UTC
Permalink
Raw Message
Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right
thing to do? Would it not be more fitting to use some git-based
command to exchange the data? For example using a git bundle? This
would then make sure that everything git knows about is transferred
into the chroot. Sbuild can make sure that git gets installed
together with build-essential and is thus able to git unbundle the
file.
What do you think?
That would be a perfectly fine answer from my point of view. However,
it has a few different behaviours. I'm not sure what is best.

Things I thought of:

1. If there are files which are ignored, or uncommitted, this would
use the committed files. I think this is a benefit.

2. Using git this way means git is definitely in the chroot.
But a package might build differently. For example, it might
have a stealth b-dep on git, which this approach wouldn't detect.
I think this is a downside of using git.

3. The git approach cannot be easily used with a different VCS.

4. Building and unpacking a git bundle is pointless work when we know
that the destination completely trusts the source, and does not
have any objects already. A tarball would do.

From the point of view of a dgit user, item 1 is quite important and
IMO outweighs the other considerations.

Regards,
Ian.
Johannes Schauer
2017-07-17 11:00:01 UTC
Permalink
Raw Message
Quoting Ian Jackson (2017-07-17 11:31:07)
Post by Ian Jackson
That would be a perfectly fine answer from my point of view. However,
it has a few different behaviours. I'm not sure what is best.
1. If there are files which are ignored, or uncommitted, this would
use the committed files. I think this is a benefit.
2. Using git this way means git is definitely in the chroot.
But a package might build differently. For example, it might
have a stealth b-dep on git, which this approach wouldn't detect.
I think this is a downside of using git.
The only assumption sbuild should make about the chroot is, that all
Essential:yes packages are installed and that apt is installed and able to
install more packages. Starting from this situation sbuild should be able to
install anything that's needed to carry out the build like build-essential. It
is not impossible to only temporarily install git for unpacking the bundle and
to then remove and autoremove it before dpkg-buildpackage is run.
Post by Ian Jackson
3. The git approach cannot be easily used with a different VCS.
The switch could be git-specific in which case it's up to the user to make sure
that the underlying VCS is really git.

If the new switch is not git-specific then sbuild could auto-detect whether
it's git (and then do git-ish stuff) or not (and then just make a tarball).
Post by Ian Jackson
4. Building and unpacking a git bundle is pointless work when we know that
the destination completely trusts the source, and does not have any
objects already. A tarball would do.
Indeed a git bundle is overkill because dpkg-buildpackage inside the chroot is
not interested in the history stored in the bundle. But just using "tar" would
not have the advantages of item 1. So "git archive" should be used together
with its disadvantages (and workaround) from item 3.
Post by Ian Jackson
From the point of view of a dgit user, item 1 is quite important and IMO
outweighs the other considerations.
Agreed.

I wonder what the best interface for this feature would be. Even more if we
consider this a long term feature, existing git workflows as well as dgit...

Thanks!

cheers, josch
Ian Jackson
2017-07-17 11:10:04 UTC
Permalink
Raw Message
Post by Johannes Schauer
Quoting Ian Jackson (2017-07-17 11:31:07)
Post by Ian Jackson
4. Building and unpacking a git bundle is pointless work when we know that
the destination completely trusts the source, and does not have any
objects already. A tarball would do.
Indeed a git bundle is overkill because dpkg-buildpackage inside the
chroot is not interested in the history stored in the bundle. But
just using "tar" would not have the advantages of item 1. So "git
archive" should be used together with its disadvantages (and
workaround) from item 3.
Post by Ian Jackson
3. The git approach cannot be easily used with a different VCS.
The switch could be git-specific in which case it's up to the user
to make sure that the underlying VCS is really git.
If the new switch is not git-specific then sbuild could auto-detect whether
it's git (and then do git-ish stuff) or not (and then just make a tarball).
That would be OK. That would leave open the possibility of using
other vcs's tarball export functions.

In this context this project came to mind:
http://www.greenend.org.uk/rjk/vcs/
Post by Johannes Schauer
I wonder what the best interface for this feature would be. Even more if we
consider this a long term feature, existing git workflows as well as dgit...
Yes. I didn't want to make some wrapper in dgit for this because
1. we already have too many wrappers in our build tools 2. my doctrine
is that dgit should be necessary only if source packages are involved.

Regards,
Ian.
Loading...