Bug#1072831: getting memory info fails when running under lxc

[Paul Slootman]

Package: procps
Version: 2:4.0.2-3
Severity: normal

I am running a number of virtual systems under lxc via libvirt.
This means these systems share the host kernel (not like qemu where a
whole virtual machine is emulated).

I upgraded one to bookworm today, and when running 'ps faxu' or 'free'
I get an error: Unable to create meminfo structure

I downgraded procps to 2:3.3.17-5 (including libprocps8 2:3.3.17-5)
and then it worked again.

Working 'free' output:

# free
total used free shared buff/cache available
Mem: 2097152 64600 1932352 9028 100200 1932352
Swap: 10338296 795392 9542904


Not working 'free' output:

# free
free: Unable to create meminfo structure


Contents of /proc/meminfo :
MemTotal: 2097152 kB
MemFree: 1930792 kB
MemAvailable: 1930792 kB
Buffers: 0 kB
Cached: 100672 kB
SwapCached: 60812 kB
Active: 119140 kB
Inactive: 38800 kB
Active(anon): 64480 kB
Inactive(anon): 84 kB
Active(file): 54660 kB
Inactive(file): 38716 kB
Unevictable: 660 kB
Mlocked: 43200 kB
SwapTotal: 10338296 kB
SwapFree: 9542904 kB
Zswap: 0 kB
Zswapped: 0 kB
Dirty: 344 kB
Writeback: 0 kB
AnonPages: 3726656 kB
Mapped: 455856 kB
Shmem: 9028 kB
KReclaimable: 142356 kB
Slab: 0 kB
SReclaimable: 0 kB
SUnreclaim: 0 kB
KernelStack: 8576 kB
PageTables: 19560 kB
SecPageTables: 48 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14295376 kB
Committed_AS: 7452164 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 42352 kB
VmallocChunk: 0 kB
Percpu: 3216 kB
HardwareCorrupted: 0 kB
AnonHugePages: 3117056 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 179704 kB
DirectMap2M: 6940672 kB
DirectMap1G: 3145728 kB


I'm willing to help debug on my system if needed.

Thanks,
Paul

-- System Information:
Debian Release: 12.5
APT prefers stable
APT policy: (800, 'stable'), (500, 'stable-updates'), (500, 'stable-security'), (500, 'unstable'), (500, 'oldstable'), (100, 'bookworm-fasttrack'), (100, 'bookworm-backports-staging')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-9-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages procps depends on:
ii init-system-helpers 1.65.2
ii libc6 2.36-9+deb12u4
ii libncursesw6 6.4-4
ii libproc2-0 2:4.0.2-3
ii libtinfo6 6.4-4

Versions of packages procps recommends:
ii psmisc 23.6-1

procps suggests no packages.

-- no debconf information

[Craig Small]

Post by Paul Slootman
# free
free: Unable to create meminfo structure

That's procps_meminfo_new() failing but /proc/meminfo exists.
The function:
checks the parameters
allocates some memory for the structure
runs meminfo_make_hash_failed(), which sets up the hashtable
runs meminfo_read failed which does some opens, seeks and reads.

Could you check to see if in the container that lxcfs has overwritten
the /proc/meminfo file? They sometimes do this for /proc/uptime. They
might have messed one of the lines up and choked procps; I'm thinking
like a tab/space at the end of the line or something that looks right
to a human but not a computer.

I think we'll need to either run a strace on free or a debugger to see
where exactly the issue is.

I'm not particularly worried about ps. This is a library meminfo issue
so fixing free should fix ps too.

procps after 3.3.17 has a *lot* of changes, but I didn't think parsing
the meminfo file was one of those.

- Craig

[Paul Slootman]

tags 1072831 patch
thanks

Post by Craig Small
Could you check to see if in the container that lxcfs has overwritten
the /proc/meminfo file? They sometimes do this for /proc/uptime. They
might have messed one of the lines up and choked procps; I'm thinking
like a tab/space at the end of the line or something that looks right
to a human but not a computer.

The /proc/meminfo in the lxc container is basically the same as that of
the host, except the total memory reflects the limit imposed on the
container. So

-MemTotal: 7914164 kB$
+MemTotal: 2097152 kB$

and the Slab / SReclaimable / SUnreclaim lines show 0 kB.

Post by Craig Small
I think we'll need to either run a strace on free or a debugger to see
where exactly the issue is.

I've been doing this.
Apparently the /proc/meminfo inside the lxc container is not seekable,
errno is ESPIPE.
The code does some seeks to reset to the beginning in preparation for
rereading the file.

I've changed the code to not do an lseek() just after opening the file
(as we should be at the start of the file then anyway), and if the file
is already open, try lseek() and if that returns errno == ESPIPE, then
close and reopen the file.

This works for me. Patch attached.

I don't know whether configure needs to test for existence of ESPIPE;
I do believe it's POSIX.


Paul

[Paul Slootman]

Post by Paul Slootman
This works for me. Patch attached.

I see I missed the case lseek() fails with another errno.
Updated patch attached.


Paul

[Jim Warner]

Hi Guys,

For what it's worth, I am unable to duplicate the meminfo problem in a
newly created debian-bookworm based lxc container.

I'm using the snap version of lxd identified as:
$ lxc --version
5.21.1 LTS

The top, ps and free programs (all identified as from procps-ng 4.0.2)
work just fine.

Regards,

Jim

[Jim Warner]

Post by Jim Warner
For what it's worth, I am unable to duplicate the meminfo problem in a
newly created debian-bookworm based lxc container.
    $ lxc --version
    5.21.1 LTS
The top, ps and free programs (all identified as from procps-ng 4.0.2)
work just fine.

Sorry, I should have also mentioned that the host is ubuntu 24.04.

[Craig Small]

Control: tags -1 fixed-upstream

Post by Paul Slootman
I see I missed the case lseek() fails with another errno.
Updated patch attached.

Thanks Paul,
This was applied upstream at
https://gitlab.com/procps-ng/procps/-/commit/104b3ce3df67092eeb868ba5e019cb895ebdf32d

We're hoping to get a new procps release soon.

- Craig

[Craig Small]

Post by Paul Slootman
I am running a number of virtual systems under lxc via libvirt.
This means these systems share the host kernel (not like qemu where a
whole virtual machine is emulated).

Hi Paul,
I did the following (as root)

lxc-create --name debtest2 --template download -- --dist debian
--release bookworm --arch amd64
sudo lxc-start --name debtest2
lxc-attach --name debtest2
***@debtest2:/# free
total used free shared buff/cache available
Mem: 32712728 13920 32694744 76 4140 32698808
Swap: 1000444 0 1000444
***@debtest2:/# free -V
free from procps-ng 4.0.2

I'm not seeing this issue at all.

[Paul Slootman]

Post by Craig Small

Post by Paul Slootman
I am running a number of virtual systems under lxc via libvirt.
This means these systems share the host kernel (not like qemu where a
whole virtual machine is emulated).

Hi Paul,
I did the following (as root)
lxc-create --name debtest2 --template download -- --dist debian
--release bookworm --arch amd64
sudo lxc-start --name debtest2
lxc-attach --name debtest2

I'm running the lxc containers under control of libvirt, I suspect the
difference is how libvirt implements the container vs. plain lxc.

With libvirt the controlling process is /usr/lib/libvirt/libvirt_lxc ,
with lxc I see /usr/bin/lxc-start .


Regards,
Paul