So, after upgrading to Bullseye, Docker is having major issues.
osmc@osmc:~$ docker run hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0001] error waiting for container: context canceled
Any ideas?
Hi
It would be good to see logs. It’s no clear what device you are even running Docker on; nor which version of OSMC you were on previously
Thanks
Sam
Ah, right you are. One day I’ll remember to include the relevant info. Apologies…
It’s a Vero 4K+, and all was working before the September update.
Log located at https://paste.osmc.tv/bonefulowi
Hm, how did you install Docker?
It looks like you have installed a Raspbian version, which probably isn’t correct.
I think the best approach is the Docker installation script: https://get.docker.com/.
Sam
I resinstalled OSMC and tried to get Docker up and running. No luck.
I’ve tried manual install and the script, both return an error worh the docker-ce package
So changed the IP Tables to legacy per this thread Can't install docker - #6 by Muenzi, I managed to manually install a Debian version of Docker. The script seems to want to install a Raspbian version
However it still will not run any images.
Just adding a “me too” so I’m on this thread – I hit the same issue immediately after upgrading to Bullseye. I was hoping to finally drop Docker for Podman but got similar looking errors with Podman too. Best guess at this point is something missing from the kernel. Still investigating; will report back if I figure anything out.
Also, just to note: I have a Docker apt source in my /etc/apt/sources.list.d that I think was originally set up by the Docker installation script:
$ cat /etc/apt/sources.list.d/download_docker_com_linux_debian.list
deb https://download.docker.com/linux/debian bullseye stable
$ dpkg -l docker-ce docker-ce-cli
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==============================-============-========================================================
ii docker-ce 5:20.10.18~3-0~debian-bullseye armhf Docker: the open-source application container engine
ii docker-ce-cli 5:20.10.18~3-0~debian-bullseye armhf Docker CLI: the open-source application container engine
Hi
Are you also on a Vero 4K/4K+?
If so – in a couple of days (bit busy right now), I can try and install Docker on a Vero and see what’s up. From what I remember last time, the installation script was treating OSMC as a Raspbian variant in error.
Cheers
Sam
Are you also on a Vero 4K/4K+?
I am, yes. Sorry, should have stated that.
Okay. I’ll give this a go when I get some time and give you an update.
Cheers
A few Google searches later and I think I’ve tracked it down to the Bullseye version of runc wanting to use “cgroup v2”, which requires kernel version 4.15 or later according to this documentation.
Kernel
Recommended version: 5.2 or later
Minimum version: 4.15Kernel older than 5.2 is not recommended due to lack of freezer.
Notably, kernel older than 4.15 MUST NOT be used (unless you are running containers with user namespaces), as it lacks support for controlling permissions of devices.
OSMC ships kernel 4.9.
Take this with a grain of salt as cgroup technical minutae is far out of my wheelhouse, but it’s a lead.
Hi Matthew,
cgroups are always evolving for me too and always complex.
I can backport this.
Isn’t there normally a Docker pre-requisite check that warns you about missing kernel CONFIG_ options? Did this flag anything up?
Cheers
Sam
From what I can see cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 being passed under our 4.9 kernel will fix this…
I’m not sure of the other implications.
Isn’t there normally a Docker pre-requisite check that warns you about missing kernel CONFIG_ options? Did this flag anything up?
From the aforementioned docs…
Am I using cgroup v2?
Yes if
/sys/fs/cgroup/cgroup.controllersis present.
And it’s present, so that may be as far as Docker checks. Indeed, docker info shows:
Server:
...
Cgroup Driver: systemd
Cgroup Version: 2
See PM re test build.
Adding a “me too” to this thread.
I had to rebuild my Vero 4K/4K+ this morning and am also getting the same error
osmc@osmc:~$ docker run hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0001] error waiting for container: context canceled
Same here :
osmc@vero:~/docker$ docker-compose up -d
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
I’m on Vero 4k
I’ve produced a test build and I’m waiting for some feedback on it.
I took a bit of time this evening to sit down with the stable release and try and reproduce the problem. Here’s what I did:
First, run
sudo update-alternatives --config iptables
and select Option 1.
Now, grab the Docker script:
wget https://get.docker.com/ -O docker.sh
It’s wrong – and thinks OSMC is Raspbian based. It’s not, so I made this change. I don’t think this is absolutely essential – but we might as well have a proper armhf version of Docker.
root@osmc:/home/osmc# diff -uar docker.sh docker_mod.sh
--- docker.sh 2022-08-29 14:27:20.000000000 +0000
+++ docker_mod.sh 2022-09-21 19:38:43.808126974 +0000
@@ -237,7 +237,7 @@
if [ -r /etc/debian_version ] && [ "$lsb_dist" != "ubuntu" ] && [ "$lsb_dist" != "raspbian" ]; then
if [ "$lsb_dist" = "osmc" ]; then
# OSMC runs Raspbian
- lsb_dist=raspbian
+ lsb_dist=debian
else
# We're Debian and don't even know it!
lsb_dist=debian
I then ran the script which installed Docker and can see it’s running:
root@osmc:/home/osmc# systemctl status docker
â—Ź docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-09-21 19:43:39 GMT; 2min 25s ago
TriggeredBy: â—Ź docker.socket
Docs: https://docs.docker.com
Main PID: 4414 (dockerd)
Tasks: 10
Memory: 13.8M
CGroup: /system.slice/docker.service
└─4414 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.417015662Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.425027661Z" level=warning msg="Unable to find cpu controller"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.425150288Z" level=warning msg="Unable to find cpuset controller"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.426372605Z" level=info msg="Loading containers: start."
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.574901497Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.744922665Z" level=info msg="Loading containers: done."
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.856523463Z" level=info msg="Docker daemon" commit=e42327a graphdriver(s)=overlay2 version=20.10.18
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.857099599Z" level=info msg="Daemon has completed initialization"
Sep 21 19:43:39 osmc systemd[1]: Started Docker Application Container Engine.
Sep 21 19:43:39 osmc dockerd[4414]: time="2022-09-21T19:43:39.995135359Z" level=info msg="API listen on /run/docker.sock"
I then tried to run a container after disabling Systemd 248 cgroup v2:
docker run hello-world
and got:
root@osmc:/home/osmc# docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(arm32v7)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
I’ll make the necessary change available shortly.
Sam
Hi
Hopefully the issue is now addressed.
I’d appreciate it if you could test this and provide feedback before we potentially release this as an update to other users. To test this update:
echo 'deb http://apt.osmc.tv bullseye-devel main' | sudo tee /etc/apt/sources.list.d/osmc-devel.list
sudo apt-get update && sudo apt-get dist-upgrade && reboot
Please see if the issue is resolved.
I also recommend you remove /etc/apt/sources.list.d/osmc-devel.list after updating.
This will deactivate the staging repository. You can do so with the following command:
sudo rm /etc/apt/sources.list.d/osmc-devel.list.
Please note that we will automatically disable this update channel after 14 days on your device in case you forget to do so to ensure that your system reverts to the stable update channel.