You can do this, but it’s not a very good idea to do so.
That would mean that we would have to optimise for the lowest common denominator.
Users would have significantly larger images due to having additional binaries they don’t need
Packaging (we have a 64bit kernel on Pi 4 and 32 bit on older platforms) via Debian wouldn’t be ideal either.
Platform deprecation would also be impossible.
This is why Raspberry Pi are ditching that one image fits all approach as well, because they were stuck with the armv6l userland for the lowest spec device (0/1)
Migrating from one Pi to another should be reasonably trivial using the built in backup tools.