Docker issues after upgrade to Bullseye

sam_nazarko · 19 September 2022 23:15

Okay. I’ll give this a go when I get some time and give you an update.

Cheers

mbarnes · 19 September 2022 23:28

A few Google searches later and I think I’ve tracked it down to the Bullseye version of runc wanting to use “cgroup v2”, which requires kernel version 4.15 or later according to this documentation.

Kernel

Recommended version: 5.2 or later
Minimum version: 4.15

Kernel older than 5.2 is not recommended due to lack of freezer.

Notably, kernel older than 4.15 MUST NOT be used (unless you are running containers with user namespaces), as it lacks support for controlling permissions of devices.

OSMC ships kernel 4.9.

Take this with a grain of salt as cgroup technical minutae is far out of my wheelhouse, but it’s a lead.

sam_nazarko · 19 September 2022 23:33

Hi Matthew,

cgroups are always evolving for me too and always complex.

I can backport this.

Isn’t there normally a Docker pre-requisite check that warns you about missing kernel CONFIG_ options? Did this flag anything up?

Cheers

Sam

sam_nazarko · 19 September 2022 23:38

From what I can see cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 being passed under our 4.9 kernel will fix this…

I’m not sure of the other implications.

mbarnes · 19 September 2022 23:42

Isn’t there normally a Docker pre-requisite check that warns you about missing kernel CONFIG_ options? Did this flag anything up?

From the aforementioned docs…

Am I using cgroup v2?

Yes if /sys/fs/cgroup/cgroup.controllers is present.

And it’s present, so that may be as far as Docker checks. Indeed, docker info shows:

Server:
 ...
 Cgroup Driver: systemd                                                                                                                                                                       
 Cgroup Version: 2

sam_nazarko · 19 September 2022 23:42

See PM re test build.

setchy · 20 September 2022 13:54

Adding a “me too” to this thread.

I had to rebuild my Vero 4K/4K+ this morning and am also getting the same error

osmc@osmc:~$ docker run hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0001] error waiting for container: context canceled

s.raynal2 · 20 September 2022 22:36

Same here :

osmc@vero:~/docker$ docker-compose up -d

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.

I’m on Vero 4k

sam_nazarko · 20 September 2022 23:56

I’ve produced a test build and I’m waiting for some feedback on it.

sam_nazarko · 21 September 2022 19:56

I took a bit of time this evening to sit down with the stable release and try and reproduce the problem. Here’s what I did:

First, run

sudo update-alternatives --config iptables

and select Option 1.

Now, grab the Docker script:

wget https://get.docker.com/ -O docker.sh

It’s wrong – and thinks OSMC is Raspbian based. It’s not, so I made this change. I don’t think this is absolutely essential – but we might as well have a proper armhf version of Docker.

root@osmc:/home/osmc# diff -uar docker.sh docker_mod.sh 
--- docker.sh	2022-08-29 14:27:20.000000000 +0000
+++ docker_mod.sh	2022-09-21 19:38:43.808126974 +0000
@@ -237,7 +237,7 @@
 			if [ -r /etc/debian_version ] && [ "$lsb_dist" != "ubuntu" ] && [ "$lsb_dist" != "raspbian" ]; then
 				if [ "$lsb_dist" = "osmc" ]; then
 					# OSMC runs Raspbian
-					lsb_dist=raspbian
+					lsb_dist=debian
 				else
 					# We're Debian and don't even know it!
 					lsb_dist=debian

I then ran the script which installed Docker and can see it’s running:

root@osmc:/home/osmc# systemctl  status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-09-21 19:43:39 GMT; 2min 25s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 4414 (dockerd)
      Tasks: 10
     Memory: 13.8M
     CGroup: /system.slice/docker.service
             └─4414 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.417015662Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.425027661Z" level=warning msg="Unable to find cpu controller"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.425150288Z" level=warning msg="Unable to find cpuset controller"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.426372605Z" level=info msg="Loading containers: start."
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.574901497Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.744922665Z" level=info msg="Loading containers: done."
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.856523463Z" level=info msg="Docker daemon" commit=e42327a graphdriver(s)=overlay2 version=20.10.18
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.857099599Z" level=info msg="Daemon has completed initialization"
Sep 21 19:43:39 osmc systemd[1]: Started Docker Application Container Engine.
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.995135359Z" level=info msg="API listen on /run/docker.sock"

I then tried to run a container after disabling Systemd 248 cgroup v2:

docker run hello-world

and got:

root@osmc:/home/osmc# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm32v7)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

I’ll make the necessary change available shortly.

Sam

sam_nazarko · 21 September 2022 20:18

Hi

Hopefully the issue is now addressed.

I’d appreciate it if you could test this and provide feedback before we potentially release this as an update to other users. To test this update:

Login via the command line
Run the following command to add the staging repository:
echo 'deb http://apt.osmc.tv bullseye-devel main' | sudo tee /etc/apt/sources.list.d/osmc-devel.list
Run the following commands to update: sudo apt-get update && sudo apt-get dist-upgrade && reboot
Your system should have have received the update.

Please see if the issue is resolved.

I also recommend you remove /etc/apt/sources.list.d/osmc-devel.list after updating.

This will deactivate the staging repository. You can do so with the following command:
sudo rm /etc/apt/sources.list.d/osmc-devel.list.

Please note that we will automatically disable this update channel after 14 days on your device in case you forget to do so to ensure that your system reverts to the stable update channel.

s.raynal2 · 21 September 2022 21:04

Hi Sam,

It does not seems to work :

osmc@vero:~/docker$ docker run hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0000] error waiting for container: context canceled

s.raynal2 · 21 September 2022 21:08

I Think I’ve done the update correctly.

Screenshot of the #2 attempt.

osmc@vero:~$ uname -a
Linux vero 4.9.269-12-osmc #1 SMP PREEMPT Wed Sep 21 12:02:25 UTC 2022 aarch64 GNU/Linux

sam_nazarko · 21 September 2022 22:50

You are not running the latest kernel.
It should be 4.9.269-13.

Please try updating again.

sam_nazarko · 22 September 2022 00:44

To go further with this, I’ve now introduced a change that will change the iptables behaviour so that new installations of OSMC after our next update will require absolutely no changes for Docker to run without issue:

For now, you will need to adjust the iptables rule; but that should be it. Whether you chose to patch the Docker script to use a more optimal version (armhf vs armv6l) is up to you, but won’t impact functionality.

Cheers

Sam

mbarnes · 22 September 2022 01:16

I can confirm the test kernel works, all my Docker containers back up and running. Thank you, Sam, for the quick resolution!

$ uname -r
4.9.269-13-osmc

$ dpkg -l docker-ce docker-ce-cli
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                        Architecture Description
+++-==============-==============================-============-========================================================
ii  docker-ce      5:20.10.18~3-0~debian-bullseye armhf        Docker: the open-source application container engine
ii  docker-ce-cli  5:20.10.18~3-0~debian-bullseye armhf        Docker CLI: the open-source application container engine

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
9b157615502d: Pull complete 
Digest: sha256:62af9efd515a25f84961b70f973a798d2eca956b1b2b026d0a4a63a3b0b6a3f2
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm32v7)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

sam_nazarko · 22 September 2022 01:17

You’re most welcome. Thanks for confirming.

The manual iptables change should also be eradicated for new installs after the next update; as covered above.

Cheers

Sam

s.raynal2 · 22 September 2022 06:28

Hi,

I re-updated, it works now !

Not sure what I did wrong last time, but it doesn’t matter.

Thanks !

Jonathan_Szoke · 22 September 2022 08:51

Confirmed, working for me too. Thanks Sam!

setchy · 24 September 2022 14:25

Thanks Sam - working perfectly!