I played with this a little, and without to much digging it looks like the pandas package is badly broken. I’d suggest you contact the pandas developers for help on this.
I did try this on Linux Mint, and
sudo apt install python3-pandas
does work. It looks like the problem is that python3-pandas-lib is missing in the debian repositories for ARM.