Discussion:
Bug#940571: dpkg-buildflags should support an optimization area for things like lto and pgo
Add Reply
Matthias Klose
2019-09-17 12:40:01 UTC
Reply
Permalink
Package: src:dpkg
Version: 1.19.7
Tags: patch

Please add an optimization area (opt, optimization) for extra optimizations like
lto (link time optimization) and pgo (profile guided optimization).

lto can be directly translated into compiler flags, as seen in the attached
patch, assuming that no lto across package boundaries is done (ensured by the
debhelper patch in #939656). The patch assumes that just nolto can be used to
disable lto until an area is introduced in dpkg.

Some upstream packages also provide build support for lto builds, so for these
an option should be given to disable the addition to the compiler flags (like
the nolto in the proposed patch).

pgo doesn't directly translate into compiler flags, but almost always requires
upstream support in the build system. pgo usually is enabled by some configure
options which are specific to the upstream build. pgo usually requires running
a profiling task, so this optimization probably should be disabled for cross
builds, otoh, the cross build then is different to the native build (although it
should create a functional identical package).
Guillem Jover
2019-10-09 18:40:01 UTC
Reply
Permalink
Control: tag -1 - patch + moreinfo
Control: retitle -1 dpkg-buildflags: Add support for an optimize area for things like LTO
Post by Matthias Klose
Package: src:dpkg
Version: 1.19.7
Tags: patch
Please add an optimization area (opt, optimization) for extra optimizations
like lto (link time optimization) and pgo (profile guided optimization).
lto can be directly translated into compiler flags, as seen in the attached
patch, assuming that no lto across package boundaries is done (ensured by
the debhelper patch in #939656). The patch assumes that just nolto can be
used to disable lto until an area is introduced in dpkg.
Some upstream packages also provide build support for lto builds, so for
these an option should be given to disable the addition to the compiler
flags (like the nolto in the proposed patch).
I take you are requesting both adding this and also enabling LTO by
default?

I'm fine with the former (even implementing that myself), but the latter
would need to go through [Q] first. And I see this can cause breakage [O]
according to the OpenSUSE people.

[Q] <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F>
[O] <https://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inter-procedural.html>
Post by Matthias Klose
pgo doesn't directly translate into compiler flags, but almost always
requires upstream support in the build system. pgo usually is enabled by
some configure options which are specific to the upstream build. pgo
usually requires running a profiling task, so this optimization probably
should be disabled for cross builds, otoh, the cross build then is different
to the native build (although it should create a functional identical
package).
I don't see how dpkg can support PGO, so I'm excluding that from this
request, as it seems this would be pretty much unactionable.

Regards,
Guillem
Matthias Klose
2019-10-10 08:50:02 UTC
Reply
Permalink
Post by Guillem Jover
Control: tag -1 - patch + moreinfo
Control: retitle -1 dpkg-buildflags: Add support for an optimize area for things like LTO
Post by Matthias Klose
Package: src:dpkg
Version: 1.19.7
Tags: patch
Please add an optimization area (opt, optimization) for extra optimizations
like lto (link time optimization) and pgo (profile guided optimization).
lto can be directly translated into compiler flags, as seen in the attached
patch, assuming that no lto across package boundaries is done (ensured by
the debhelper patch in #939656). The patch assumes that just nolto can be
used to disable lto until an area is introduced in dpkg.
Some upstream packages also provide build support for lto builds, so for
these an option should be given to disable the addition to the compiler
flags (like the nolto in the proposed patch).
I take you are requesting both adding this and also enabling LTO by
default?
the infrastructure should provide both, having the option to enable it by a flag
when it's not the default, and to disable it by default, when it's enabled by
default.
Post by Guillem Jover
I'm fine with the former (even implementing that myself), but the latter
would need to go through [Q] first. And I see this can cause breakage [O]
according to the OpenSUSE people.
[Q] <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F>
[O] <https://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inter-procedural.html>
well, yes, it breaks a few packages, more than a handful, but not the whole
archive, and you can name these packages. The question for now is how to name
that option, and how to disable that on a per package basis. And we probably
want to enable that for a subset of architectures first, which have been test
built. So the first thing is the name, so people can disable lto, before we
even consider making it the default.

It's now on by default in Suse and Fedora, and I'm evaluating a complete Ubuntu
test rebuild with lto on. So there are a few things which need to be manually
disabled, so I'm asking about the naming first. I don't think that lto by
default is out of scope for bullseye.
Post by Guillem Jover
Post by Matthias Klose
pgo doesn't directly translate into compiler flags, but almost always
requires upstream support in the build system. pgo usually is enabled by
some configure options which are specific to the upstream build. pgo
usually requires running a profiling task, so this optimization probably
should be disabled for cross builds, otoh, the cross build then is different
to the native build (although it should create a functional identical
package).
I don't see how dpkg can support PGO, so I'm excluding that from this
request, as it seems this would be pretty much unactionable.
The only thing it would do is to provide a common interface to enable/disable
it, not an implementation.
Guillem Jover
2019-10-12 04:20:01 UTC
Reply
Permalink
Post by Matthias Klose
Post by Guillem Jover
I take you are requesting both adding this and also enabling LTO by
default?
the infrastructure should provide both, having the option to enable it by a
flag when it's not the default, and to disable it by default, when it's
enabled by default.
This is something provided for all supported features, but this does
not answer whether you are requesting this to be the default or not.

Or is your plan to enable this by default in gcc instead of via flags
passed by dpkg-buildflags?
Post by Matthias Klose
Post by Guillem Jover
I'm fine with the former (even implementing that myself), but the latter
would need to go through [Q] first. And I see this can cause breakage [O]
according to the OpenSUSE people.
[Q] <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F>
[O] <https://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inter-procedural.html>
well, yes, it breaks a few packages, more than a handful, but not the whole
archive, and you can name these packages. The question for now is how to
name that option, and how to disable that on a per package basis. And we
probably want to enable that for a subset of architectures first, which have
been test built. So the first thing is the name, so people can disable lto,
before we even consider making it the default.
If you are suggesting repeating the same disaster that we have with
pie, with the default set by gcc being arch opt-in, then I'm honestly
less than interested in doing the same here, and I'd consider this a
direct wontfix. :/
Post by Matthias Klose
Post by Guillem Jover
Post by Matthias Klose
pgo doesn't directly translate into compiler flags, but almost always
requires upstream support in the build system. pgo usually is enabled by
some configure options which are specific to the upstream build. pgo
usually requires running a profiling task, so this optimization probably
should be disabled for cross builds, otoh, the cross build then is different
to the native build (although it should create a functional identical
package).
I don't see how dpkg can support PGO, so I'm excluding that from this
request, as it seems this would be pretty much unactionable.
The only thing it would do is to provide a common interface to
enable/disable it, not an implementation.
Ok, I guess, given that using unknown features for known areas emits
warnings (even though it seems weird and unexpected to support a feature
for which dpkg-buildflags will never be able to emit flags for, and some
other name could be used for this, but consistency would be nice).

Regards,
Guillem
Matthias Klose
2019-10-12 17:40:02 UTC
Reply
Permalink
Post by Guillem Jover
Post by Matthias Klose
Post by Guillem Jover
I take you are requesting both adding this and also enabling LTO by
default?
the infrastructure should provide both, having the option to enable it by a
flag when it's not the default, and to disable it by default, when it's
enabled by default.
This is something provided for all supported features, but this does
not answer whether you are requesting this to be the default or not.
Not yet. But yes, it should be an option for bullseye.
Post by Guillem Jover
Or is your plan to enable this by default in gcc instead of via flags
passed by dpkg-buildflags?
No, optimization flags are not turned on by default in GCC.
Post by Guillem Jover
Post by Matthias Klose
Post by Guillem Jover
I'm fine with the former (even implementing that myself), but the latter
would need to go through [Q] first. And I see this can cause breakage [O]
according to the OpenSUSE people.
[Q] <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F>
[O] <https://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inter-procedural.html>
well, yes, it breaks a few packages, more than a handful, but not the whole
archive, and you can name these packages. The question for now is how to
name that option, and how to disable that on a per package basis. And we
probably want to enable that for a subset of architectures first, which have
been test built. So the first thing is the name, so people can disable lto,
before we even consider making it the default.
If you are suggesting repeating the same disaster that we have with
pie, with the default set by gcc being arch opt-in, then I'm honestly
less than interested in doing the same here, and I'd consider this a
direct wontfix. :/
Post by Matthias Klose
Post by Guillem Jover
Post by Matthias Klose
pgo doesn't directly translate into compiler flags, but almost always
requires upstream support in the build system. pgo usually is enabled by
some configure options which are specific to the upstream build. pgo
usually requires running a profiling task, so this optimization probably
should be disabled for cross builds, otoh, the cross build then is different
to the native build (although it should create a functional identical
package).
I don't see how dpkg can support PGO, so I'm excluding that from this
request, as it seems this would be pretty much unactionable.
The only thing it would do is to provide a common interface to
enable/disable it, not an implementation.
Ok, I guess, given that using unknown features for known areas emits
warnings (even though it seems weird and unexpected to support a feature
for which dpkg-buildflags will never be able to emit flags for, and some
other name could be used for this, but consistency would be nice).
CC'ing Holger here, as this came up with reproducible builds. PGO and
reproducible builds are currently a no-go. So having a nopgo flag would enable
testing reproducible builds without pgo, if they succeed or not.

Mathias

Bernhard M. Wiedemann
2019-10-10 12:00:01 UTC
Reply
Permalink
LTO:
ensure you have that gcc patch
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91307

There might be few other related patches that our (SUSE's) Martin Liska
probably knows, but if you already have the -flto=auto, you probably got
the others, too.
Post by Matthias Klose
The only thing it would do is to provide a common interface to enable/disable
it, not an implementation.
In openSUSE, we have a boolean %do_profiling RPM define
that can be used in individual .spec files, for example

https://github.com/bmwiedemann/openSUSE/blob/ca729bd0021b6ed6793807158bb3aabadbde4a4b/packages/b/bash/bash.spec#L418

Overall, it is still very hard to get reproducible builds with PGO
enabled, but you can see gzip and libsamplerate examples in
https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/pgo
for places where it was possible.
Sometimes it might make sense to have a 2nd pgo value that answers the
question if performance is more important than reproducible build results.
Loading...