Also got it working:
just had to update the arch in the sources list file at /etc/apt/sources.list.d/docker.list
to
deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bullseye stable
purge the docker packages, update apt and the reinstall. No need to manually install deb files. Get this error whenever running docker containers, but I can live with it
ERROR: ld.so: object '/usr/lib/libarmmem.so' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.