It depends entirely on the volume level in the file - which varies greatly from file to file and also depending on whether you are trying to downmix from 5.1 to stereo or not.
On a file that is already stereo and has had compression applied during ripping the audio will be fairly loud most of the time so you probably won’t notice an issue.
But play a file with 5.1 audio downmixed to stereo on the Pi and you’ll get a big loss in volume level, now you’ll start to notice noise and distortion on the analog output.