Discussion:
Bug#925899: lxc: Unprivileged containers fail to start after recent updates
(too old to reply)
Regis Smith
2019-03-28 05:30:01 UTC
Permalink
Package: lxc
Version: 1:3.1.0+really3.0.3-6
Severity: important

Dear Maintainer,

* What led up to the situation?

apt update; apt upgrade

* What exactly did you do (or not do) that was effective (or
ineffective)?

As a normal user:
$ lxc-start -n test

* What was the outcome of this action?

lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 833 No such file or directory - Failed to receive the container state
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

If I run it in the foreground instead I get

$ lxc-start -n test -F
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use generated profile: apparmor_parser not available
lxc-start: test: start.c: lxc_init: 899 Failed to initialize LSM
lxc-start: test: start.c: __lxc_start: 1917 Failed to initialize container "test"
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

* What outcome did you expect instead?

A running container. These used to work up until recently. Now I can't stop
already running containers because I won't be able to restart them.

-- System Information:
Debian Release: buster/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages lxc depends on:
ii debconf [debconf-2.0] 1.5.71
ii libc6 2.28-8
ii libcap2 1:2.25-2
ii libgnutls30 3.6.6-2
ii liblxc1 1:3.1.0+really3.0.3-6
ii libseccomp2 2.3.3-4
ii libselinux1 2.8-1+b1
ii lsb-base 10.2019031300

Versions of packages lxc recommends:
ii bridge-utils 1.6-2
ii debootstrap 1.0.114
ii dirmngr 2.2.12-1
ii dnsmasq-base [dnsmasq-base] 2.80-1
ii gnupg 2.2.12-1
ii iproute2 4.20.0-2
ii iptables 1.8.2-4
ii libpam-cgfs 1:3.1.0+really3.0.3-6
ii lxc-templates 3.0.3-1
ii lxcfs 3.0.3-2
ii nftables 0.9.0-2
ii openssl 1.1.1b-1
ii rsync 3.1.3-6
ii uidmap 1:4.5-1.1

Versions of packages lxc suggests:
ii apparmor 2.13.2-9
ii btrfs-progs 4.20.1-2
ii lvm2 2.03.02-2
pn python3-lxc <none>

-- Configuration Files:
/etc/lxc/default.conf changed:
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1


-- debconf information:
* lxc/auto_update_config: true
Pierre-Elliott Bécue
2019-03-30 14:10:01 UTC
Permalink
Post by Regis Smith
Package: lxc
Version: 1:3.1.0+really3.0.3-6
Severity: important
Dear Maintainer,
* What led up to the situation?
apt update; apt upgrade
* What exactly did you do (or not do) that was effective (or
ineffective)?
$ lxc-start -n test
* What was the outcome of this action?
lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 833 No such file or directory - Failed to receive the container state
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
If I run it in the foreground instead I get
$ lxc-start -n test -F
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use generated profile: apparmor_parser not available
lxc-start: test: start.c: lxc_init: 899 Failed to initialize LSM
lxc-start: test: start.c: __lxc_start: 1917 Failed to initialize container "test"
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
* What outcome did you expect instead?
A running container. These used to work up until recently. Now I can't stop
already running containers because I won't be able to restart them.
Hi,

Thanks for submitting this bug.

As you can see, it is possible to get more debug via the --logfile and
the --logpriority options.
Post by Regis Smith
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
It means that you're lacking the apparmor_parser command, which is
shipped by apparmor. It probably means that you refused to install
apparmor on your host.

You have multiple choices. The first one being installing apparmor, and
the second one being to edit your container's configuration (or the
/etc/lxc/default.conf file) to change the lxc.apparmor.profile
parameter.

This bugreport raises an interesting question regarding the tradeoff
between the solution we implemented to fix bug #916639.

Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their configuration.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916639
--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528 F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.
intrigeri
2019-03-30 14:40:01 UTC
Permalink
Hi,
Post by Pierre-Elliott Bécue
This bugreport raises an interesting question regarding the tradeoff
between the solution we implemented to fix bug #916639.
Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their configuration.
I'd love to help but I'll need more info to understand why the current
setup breaks "many unpriv containers", e.g.:

- Is this specific to unprivileged containers?

- Is it because "apparmor.profile = generated" is not suitable
for unprivileged containers?

Finally, I wonder if "Suggests: apparmor" expresses strongly enough
the current status of the LXC + AppArmor integration in Debian.
Thankfully the Linux images will pull apparmor via Recommends…
except on systems where the administrator has disabled installation
of Recommends.

Cheers,
--
intrigeri
Pierre-Elliott Bécue
2019-03-30 15:10:02 UTC
Permalink
Post by Pierre-Elliott Bécue
Hi,
Post by Pierre-Elliott Bécue
This bugreport raises an interesting question regarding the tradeoff
between the solution we implemented to fix bug #916639.
Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their
configuration.
I'd love to help but I'll need more info to understand why the current
- Is this specific to unprivileged containers?
- Is it because "apparmor.profile = generated" is not suitable
for unprivileged containers?
Finally, I wonder if "Suggests: apparmor" expresses strongly enough
the current status of the LXC + AppArmor integration in Debian.
Thankfully the Linux images will pull apparmor via Recommends…
except on systems where the administrator has disabled installation
of Recommends.
Cheers,
It is specific to unpriviledged containers and due to the fact that non root users don't seem to have the ability to use the generated profile.
PEB (from my phone)
Regis Smith
2019-03-30 18:40:02 UTC
Permalink
Hi! Thanks for replying.

On Sat, 30 Mar 2019 14:51:47 +0100 Pierre-Elliott =?utf-8?B?QsOpY3Vl?=
Post by Pierre-Elliott Bécue
Post by Regis Smith
Package: lxc
Version: 1:3.1.0+really3.0.3-6
Severity: important
Dear Maintainer,
* What led up to the situation?
apt update; apt upgrade
* What exactly did you do (or not do) that was effective (or
ineffective)?
$ lxc-start -n test
* What was the outcome of this action?
lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 833 No
such file or directory - Failed to receive the container state
Post by Pierre-Elliott Bécue
Post by Regis Smith
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 333 To get more details,
run the container in foreground mode
Post by Pierre-Elliott Bécue
Post by Regis Smith
lxc-start: test: tools/lxc_start.c: main: 336 Additional
information can be obtained by setting the --logfile and --logpriority
options
Post by Pierre-Elliott Bécue
Post by Regis Smith
If I run it in the foreground instead I get
$ lxc-start -n test -F
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
Post by Pierre-Elliott Bécue
Post by Regis Smith
lxc-start: test: start.c: lxc_init: 899 Failed to initialize LSM
lxc-start: test: start.c: __lxc_start: 1917 Failed to initialize container "test"
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 336 Additional
information can be obtained by setting the --logfile and --logpriority
options
Post by Pierre-Elliott Bécue
Post by Regis Smith
* What outcome did you expect instead?
A running container. These used to work up until recently. Now I can't stop
already running containers because I won't be able to restart them.
Hi,
Thanks for submitting this bug.
As you can see, it is possible to get more debug via the --logfile and
the --logpriority options.
Post by Regis Smith
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
It means that you're lacking the apparmor_parser command, which is
shipped by apparmor. It probably means that you refused to install
apparmor on your host.
Actually, I do have apparmor installed, and I can run apparmor_parser
as root. aa-status shows all the related "lxc-container-*" in enforce
mode. Priveleged containers work fine, but I can not start unprivileged
containers. Both privileged and unpriveleged worked fine before the
updates over the past several weeks.
Post by Pierre-Elliott Bécue
You have multiple choices. The first one being installing apparmor, and
the second one being to edit your container's configuration (or the
/etc/lxc/default.conf file) to change the lxc.apparmor.profile
parameter.
This bugreport raises an interesting question regarding the tradeoff
I attached the log from running

$ lxc-start -n test --logpriority DEBUG --logfile lxc.log

I commented out "apparmor.profile = generated" and it still doesn't
work. I'd like to get this working with apparmor, since it's the
default. However, I'd love to hear from anyone who has unprivileged
containers working on an up-to-date Buster. The fickleness of LXC in
Stretch wore me out, so I was quite pleased when it worked reliably in
Buster, up until now.

Regis
intrigeri
2019-03-31 13:00:02 UTC
Permalink
Hi,
Post by Regis Smith
Post by Regis Smith
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
I've reproduced this problem and I could fix it with:

lxc.apparmor.profile = unconfined

Regis, can you please confirm this fix works for you as well?
Post by Regis Smith
Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their configuration.
Considering kernel.unprivileged_userns_clone is disabled by default
on Debian, IMO we should:

- Optimize for the Debian defaults, i.e. privileged containers:
- Keep the settings we added recently in /etc/lxc/default.conf
- Replace "Suggests: apparmor" with "Depends: apparmor", because
the default config will create containers that fail to start
if the apparmor package is not installed.

- Document how to use unprivileged containers on Debian. It's not as
if they were previously working fine by default and AppArmor broke
them — regardless of AppArmor, on current sid with the default
kernel settings and lxc.apparmor.profile = unconfined, trying to
start an unprivileged container fails in a very much user
unfriendly way:

conf.c: chown_mapped_root: 3250 lxc-usernsexec failed: Permission denied - Failed to open tt

That's a first usability stumbling block. The new
lxc.apparmor.profile default setting merely adds a second one.

So I think README.Debian should document the need for
kernel.unprivileged_userns_clone=1 and for
lxc.apparmor.profile = unconfined

- Take care of the Stretch→Buster upgrade path for unprivileged
containers, by mentioning in NEWS.Debian that previously working
unprivileged containers now need lxc.apparmor.profile = unconfined.

Thoughts?

Cheers,
--
intrigeri
Pierre-Elliott Bécue
2019-03-31 14:20:02 UTC
Permalink
Post by intrigeri
Post by Pierre-Elliott Bécue
Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their configuration.
Considering kernel.unprivileged_userns_clone is disabled by default
- Keep the settings we added recently in /etc/lxc/default.conf
- Replace "Suggests: apparmor" with "Depends: apparmor", because
the default config will create containers that fail to start
if the apparmor package is not installed.
- Document how to use unprivileged containers on Debian. It's not as
if they were previously working fine by default and AppArmor broke
them — regardless of AppArmor, on current sid with the default
kernel settings and lxc.apparmor.profile = unconfined, trying to
start an unprivileged container fails in a very much user
conf.c: chown_mapped_root: 3250 lxc-usernsexec failed: Permission denied - Failed to open tt
That's a first usability stumbling block. The new
lxc.apparmor.profile default setting merely adds a second one.
So I think README.Debian should document the need for
kernel.unprivileged_userns_clone=1 and for
lxc.apparmor.profile = unconfined
Yes and no, the issue is that we introduced a new problem which made the
unpriviledged containers of user having already set the kernel properly
not working either.

There is a tradeoff in your current solution, and I wonder if it is the
appropriate one.

I'm Cc-ing Antonio and Evgeni to get their POV.
Post by intrigeri
- Take care of the Stretch→Buster upgrade path for unprivileged
containers, by mentioning in NEWS.Debian that previously working
unprivileged containers now need lxc.apparmor.profile = unconfined.
The apparmor.profile = lxc-container-default-cgns works too, but will
prevent systemd isolation features to work.
--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528 F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.
Regis Smith
2019-03-31 20:30:02 UTC
Permalink
Post by intrigeri
Hi,
Post by Regis Smith
Post by Regis Smith
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
lxc.apparmor.profile = unconfined
Regis, can you please confirm this fix works for you as well?
Yes, this works. Thanks! This was the old solution required in Jessie
or early Stretch (w/o backports), so people may still complain after
Buster is released :) But I'm happpy if I can reliably start
unprivileged containers again, which "unconfined" does.

Regis
Pierre-Elliott Bécue
2019-04-07 18:40:01 UTC
Permalink
Post by intrigeri
Hi,
Post by Regis Smith
Post by Regis Smith
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
lxc.apparmor.profile = unconfined
Regis, can you please confirm this fix works for you as well?
Post by Regis Smith
Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their configuration.
Considering kernel.unprivileged_userns_clone is disabled by default
- Keep the settings we added recently in /etc/lxc/default.conf
- Replace "Suggests: apparmor" with "Depends: apparmor", because
the default config will create containers that fail to start
if the apparmor package is not installed.
- Document how to use unprivileged containers on Debian. It's not as
if they were previously working fine by default and AppArmor broke
them — regardless of AppArmor, on current sid with the default
kernel settings and lxc.apparmor.profile = unconfined, trying to
start an unprivileged container fails in a very much user
conf.c: chown_mapped_root: 3250 lxc-usernsexec failed: Permission denied - Failed to open tt
That's a first usability stumbling block. The new
lxc.apparmor.profile default setting merely adds a second one.
So I think README.Debian should document the need for
kernel.unprivileged_userns_clone=1 and for
lxc.apparmor.profile = unconfined
- Take care of the Stretch→Buster upgrade path for unprivileged
containers, by mentioning in NEWS.Debian that previously working
unprivileged containers now need lxc.apparmor.profile = unconfined.
Thoughts?
See the two latest commits for lxc:

https://salsa.debian.org/lxc-team/lxc/commits/master

Tell me what you think about them, and if needed don't hesitate to
submit a MR! :)
--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528 F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.
intrigeri
2019-04-08 07:10:01 UTC
Permalink
Hi,
Post by Pierre-Elliott Bécue
https://salsa.debian.org/lxc-team/lxc/commits/master
Tell me what you think about them,
Sweet, thanks! :)
Post by Pierre-Elliott Bécue
and if needed don't hesitate to submit a MR! :)
There you go: https://salsa.debian.org/lxc-team/lxc/merge_requests/4

Cheers,
--
intrigeri
Loading...