Libc6 dependencies problem after update to 2.28-110.1?

On clean installation on my rpi4 ( [2021.08-1]) and upgrade via ‘sudo apt-get update && sudo apt-get -y dist-upgrade’ which upgrading libc6 and some other things, I cannot install docker due to conflict in dependencies:

osmc@osmc:~$ sudo apt-get install docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 containerd.io:arm64 : Depends: libc6:arm64 (>= 2.17) but it is not going to be installed
                       Depends: libseccomp2:arm64 (>= 2.3.0) but it is not going to be installed
 docker-ce:arm64 : Depends: libseccomp2:arm64 (>= 2.3.0) but it is not going to be installed
                   Depends: libc6:arm64 (>= 2.17) but it is not going to be installed
                   Depends: libdevmapper1.02.1:arm64 (>= 2:1.02.97) but it is not going to be installed
                   Depends: libsystemd0:arm64 but it is not going to be installed
 docker-ce-cli:arm64 : Depends: libc6:arm64 (>= 2.17) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
osmc@osmc:~$

On the other hand, in rpi4 installation where docker is already installed, on ‘sudo apt-get update && sudo apt-get -y dist-upgrade’ I’m getting “The following packages have been kept back: libc6”

When I try to dry-run to manually update libc6 I’m getting:

$ sudo apt-get --dry-run install libc6
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  gcc-8-base:arm64 linux-libc-dev
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  glibc-doc
The following packages will be REMOVED:
  containerd.io:arm64 docker-ce:arm64 docker-ce-cli:arm64 libblkid1:arm64 libc6:arm64 libdevmapper1.02.1:arm64
  libgcc1:arm64 libgcrypt20:arm64 libgpg-error0:arm64 liblz4-1:arm64 liblzma5:arm64 libpcre3:arm64 libseccomp2:arm64
  libselinux1:arm64 libsystemd0:arm64 libudev1:arm64 libuuid1:arm64
The following packages will be upgraded:
  libc6
1 upgraded, 0 newly installed, 17 to remove and 0 not upgraded.
Remv docker-ce:arm64 [5:20.10.8~3-0~debian-buster]
Remv containerd.io:arm64 [1.4.9-1]
Remv docker-ce-cli:arm64 [5:20.10.8~3-0~debian-buster]
Remv libdevmapper1.02.1:arm64 [2:1.02.155-3]
Remv libblkid1:arm64 [2.33.1-0.1]
Remv libudev1:arm64 [241-7~deb10u8]
Remv libsystemd0:arm64 [241-7~deb10u8]
Remv libc6:arm64 [2.28-10] [libseccomp2:arm64 libgpg-error0:arm64 libgcc1:arm64 libpcre3:arm64 libuuid1:arm64 libgcrypt20:arm64 libselinux1:arm64 liblz4-1:arm64 liblzma5:arm64 ]
Remv libgcc1:arm64 [1:8.3.0-6] [libseccomp2:arm64 libgpg-error0:arm64 libpcre3:arm64 libuuid1:arm64 libgcrypt20:arm64 libselinux1:arm64 liblz4-1:arm64 liblzma5:arm64 ]
Remv libgcrypt20:arm64 [1.8.4-5+deb10u1] [libseccomp2:arm64 libgpg-error0:arm64 libpcre3:arm64 libuuid1:arm64 libselinux1:arm64 liblz4-1:arm64 liblzma5:arm64 ]
Remv libgpg-error0:arm64 [1.35-1] [libseccomp2:arm64 libpcre3:arm64 libuuid1:arm64 libselinux1:arm64 liblz4-1:arm64 liblzma5:arm64 ]
Remv liblz4-1:arm64 [1.8.3-1+deb10u1] [libseccomp2:arm64 libpcre3:arm64 libuuid1:arm64 libselinux1:arm64 liblzma5:arm64 ]
Remv liblzma5:arm64 [5.2.4-1] [libseccomp2:arm64 libpcre3:arm64 libuuid1:arm64 libselinux1:arm64 ]
Remv libselinux1:arm64 [2.8-1+b1] [libseccomp2:arm64 libpcre3:arm64 libuuid1:arm64 ]
Remv libpcre3:arm64 [2:8.39-12] [libseccomp2:arm64 libuuid1:arm64 ]
Remv libseccomp2:arm64 [2.3.3-4] [libuuid1:arm64 ]
Remv libuuid1:arm64 [2.33.1-0.1]
Inst libc6 [2.28-10] (2.28-110.1 OSMC:apt.osmc.tv [armhf])
Conf libc6 (2.28-110.1 OSMC:apt.osmc.tv [armhf])
$

Meaning docker will be removed (and it removes indeed, I tried, and then it goes back to impossibility to install, see beginning)

Please advise…

Like the Vero4k/+, the Pi 4 uses a multi-architecture set-up, with a 64-bit kernel and 32-bit userland.

You haven’t explained how you reached a situation where you can remove 64-bit userland libraries, though I suspect it’ll be in the APT sources list. But to answer your main question, you might have more success with 32-bit (armhf) docker.

I’m not sure I follow…

Before last upgrade of libc6 everything was (and still is on my rpi4-osmc machine) fine - 64bit, AFAIU:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)

Server:
 Containers: 10
  Running: 10
  Paused: 0
  Stopped: 0
 Images: 19
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.32-1-osmc
 Operating System: Open Source Media Center
 OSType: linux
Architecture: aarch64
 CPUs: 4
 Total Memory: 3.705GiB

This is actually first time when this problem occurs.

As for the second thing - I’m not quite sure what you’re asking about removing 64bit libs…

Sorry, I misread your post, but my original point about the Pi 4 being multi-architecture still stands. If you run:

sudo apt-get --dry-run install libc6

it’ll install the armhf version. If you need the arm64 version, it should be:

osmc@osmc:~$ sudo apt-get --dry-run install libc6:arm64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gcc-8-base:arm64 libgcc1:arm64
Suggested packages:
  glibc-doc:arm64 locales:arm64
Recommended packages:
  libidn2-0:arm64
The following NEW packages will be installed:
  gcc-8-base:arm64 libc6:arm64 libgcc1:arm64
0 upgraded, 3 newly installed, 0 to remove and 2 not upgraded.
Inst gcc-8-base:arm64 (8.3.0-6 Debian:10.10/oldstable [arm64])
Inst libgcc1:arm64 (1:8.3.0-6 Debian:10.10/oldstable [arm64]) []
Inst libc6:arm64 (2.28-10 Debian:10.10/oldstable [arm64])
Conf gcc-8-base:arm64 (8.3.0-6 Debian:10.10/oldstable [arm64])
Conf libgcc1:arm64 (1:8.3.0-6 Debian:10.10/oldstable [arm64])
Conf libc6:arm64 (2.28-10 Debian:10.10/oldstable [arm64])

Perhaps it would also be better if you provide logs grab-logs -A from both machines. At least it’ll be clearer what’s on your systems.

Thanks for your help!

So, I have Machine A: rpi4/2Gb, fresh install + dist-upgrade, then docker install (as you suggested, I’ve installed 32bit, armhf, with success): https://paste.osmc.tv/decijuyuqo

Machine B: rpi4/4Gb, docker (64bit, AFAIU) installed BEFORE last dist-upgrade which cannot upgrade libc6 (The following packages have been kept back): https://paste.osmc.tv/iliwabigus
I tried on Machine B:

~$ sudo apt-get --dry-run install libc6:arm64
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6:arm64 is already the newest version (2.28-10).
libc6:arm64 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

Which is correct, because OSMC wants it to be 2.28-110, and docker cannot comply

I guess, my question is: is everything fine with last upgrade?.. Maybe I just need to wait for the next one, where everything will be fine again?.. Will it somehow bite me in the ass? :slight_smile:

Ok, I think I see a problem.

The default Debian version of libc6 (both for armhf and arm64) is 2.28-10.

For some reason, the OSMC repo now contains a higher version 2.28-110 of libc6 (and other glibc libs) but only for armhf.

One of the rules of Debian multiarch is that versions must match exactly across architectures, which is not the case here, so this is a problem created by @sam_nazarko.

Your libc6 libraries are both at 2.28-10:

ii  libc6:arm64                          2.28-10                             arm64        GNU C Library: Shared libraries
ii  libc6:armhf                          2.28-10                             armhf        GNU C Library: Shared libraries

so you have no need to update libc6 (in fact, you can’t). If you need to run a Debian dist-upgrade, I’d suggest that you comment out the OSMC repo for now and run an apt-get update / dist-upgrade. Alternatively, wait for Sam to fix the repo.

1 Like

To support Widevine changes.

Not something we will fix.

ARM64 and any building should be in toolchain ideally.

To support multiarch, you need to have an arm64 version of libc6 in the OSMC repo.

This is nothing to do with the toolchain. As I noted above:

This rule applies for any package where the OSMC repo has a higher package version than is found in the Debian repo. Because you have added a bunch of 2,28-110 glibc libraries to pool/main/g/glibc, all these armhf packages need to have arm64 equivalents in the OSMC repo with the same version number.

1 Like

That’s what I’m saying – there’s no need for us to build an arm64 version IMHO.
User should use our toolchain instead – because we are armhf first, and aarch64 is only included for kernel support packages.

We don’t care about or support Aarch64 userland at this stage.

@sam_nazarko does it mean that I need to uninstall docker/arm64, update libc6 and install docker/armhf? no other way around it?..

The thing is I’m starting to have problems on docker/armhf with images as well - for example, linuxserver/wireguard stopped working.

Uname info: Linux 7c9386b426c6 5.10.32-1-osmc #1 SMP PREEMPT Wed Apr 28 05:45:57 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
**** It seems the wireguard module is already active. Skipping kernel header install and module compilation. ****
**** Client mode selected. ****
**** Disabling CoreDNS ****

It looks like system wireguard drivers is arm64, but docker/32bit with image seems to not understand what is going on (?)

Could you please return compatibility that was before last update - or at least advise how to proceed?..
Thanks in advance

Run apt-get install aarch64-toolchain-osmc
Build anything you want to be ARM64 in that toolchain environment.

I don’t know why you specifically need ARM64 binaries in your environment. If there’s a use case, I can look in to it further.

@sam_nazarko I’m not sure how exactly I can install docker/arm64 after installing aarch64-toolchain-osmc - it still gives me dependency problems:

$ sudo apt-get install docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 containerd.io:arm64 : Depends: libc6:arm64 (>= 2.17) but it is not going to be installed
                       Depends: libseccomp2:arm64 (>= 2.3.0) but it is not going to be installed
 docker-ce:arm64 : Depends: libseccomp2:arm64 (>= 2.3.0) but it is not going to be installed
                   Depends: libc6:arm64 (>= 2.17) but it is not going to be installed
                   Depends: libdevmapper1.02.1:arm64 (>= 2:1.02.97) but it is not going to be installed
                   Depends: libsystemd0:arm64 but it is not going to be installed
 docker-ce-cli:arm64 : Depends: libc6:arm64 (>= 2.17) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

My first use case I presented above: I want to run docker container with wireguard VPN (Docker), and if I’m using docker/armhf installation it apparently tries to use system’s module (aarch64), which breaks it completely - no traffic at all.

Uname info: Linux 7c9386b426c6 5.10.32-1-osmc #1 SMP PREEMPT Wed Apr 28 05:45:57 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
**** It seems the wireguard module is already active. Skipping kernel header install and module compilation. ****
**** Client mode selected. ****
**** Disabling CoreDNS ****

I’m not sure how can I solve this, and I fear this isn’t the only one artefact of this arm64/armhf duality which I will face.

For now, I’m on libc 2.28.10 and everything works fine with arm64. It would be really unfortunate if I lose further updates to OSMC because of that…

I really don’t understand your position.

Multiarch is a capability that is part of the Debian OS and is fully supported by Debian.

By adding glibc libraries only for the armhf architecture you have broken Debian.

You have broken Debian for the benefit of Widevine users, but at the expense of multiarch users. This was done without consultation or debate. Is there any valid reason why Widevine is prioritised over multiarch – and isn’t this something that’s worthy of an open discussion?

I’m guessing that it was a mistake in that you didn’t anticipate the consequences of adding just armhf libraries. Fair enough. Stuff happens.

What I don’t understand is your reluctance to fix the mistake. It is not the responsibility of OSMC users to justify why you should unbreak Debian, especially since the fix is so simple.

1 Like

We can add glibc support for Aarch64 as well, but the point I’m trying to make clear is that OSMC doesn’t really support Aarch64 userland. Support won’t come until we have a target that is both Aarch64 kernel + Aarch64 userland entirely, without armhf.

We tried this in 2017, but upstream Debian had too many problems and I’d still argue that Debian isn’t ready yet – i.e. Debian is broken already for this architecture. Moving to Aarch64 only also has its own problems from a Widevine perspective (at least for Pi platforms).

But we shouldn’t build packages for MIPS etc too.

I do agree however that user’s should be able to install ARM64 packages via APT and will get that solved, so thanks for bringing it to my attention. But I don’t expect us to support this: i.e. if these packages misbehave, then you are on your own

Cheers

Sam

This should now be fixed – at least for OP’s use case.

@dillthedog thanks for your support!

@sam_nazarko thanks, everything’s working now on 2.28.110, including docker+wireguard/arm64 and amazon/netflix via Widevine

Thanks for confirming.

You can also thank @dillthedog for his advice too!