The volume control in this type of setup is adjusting how loud of a signal Kodi is sending out. The optical connection has no signaling for volume as part of its spec so Kodi can only control its side of the volume. As such the volume you hear is a function of both Kodi, and your soundbars volume. If Kodi’s volume is low and your soundbar is all the way up, then it will be low. If Kodi is all the way up but your soundbar is set at a low volume then it will be low. The way one would normally set this up is to turn Kodi to max volume and then turn the soundbar up to where it is as loud as you want it to get.
This may be a bit more complicated now as something in Kodi seems to have changed such that its volume gets reset back to 100% when it is rebooted. There is a fix to change this I found and outlined [here]
Things work a bit better with a HDMI connected soundbar as Kodi will switch over from controlling its volume via adjusted its output volume to directly controlling the volume on the soundbar itself via CEC. This also gives a bit of a gain in audio quality as you get a better S/N ratio outputting at full output from Kodi.
There is an additional complication to how you have your settings at the moment. Kodi will only adjust the volume of a PCM signal. If you have passthrough turned on for AC3 and DTS then anything with these particular codecs will only output at full volume from Kodi, period, full stop. When you are passing through these formats that are in the original files then adjusting Kodi’s volume should show up on the screen as an icon, but the volume doesn’t change as Kodi will not touch what is passing through. What the inconsistency your likely seeing is if your playing mp3, FLAC, AAC, PCM, DTS MA, eAC3, etc. then Kodi is converting them to PCM where it can control the volume, and then because of your settings it is then converting it to an AC3 file that then has modified volume.
With your hardware setup you need to just output stereo and turn off passthrough, or else give up on controlling volume from Kodi. If you wanted to enable passthrough in case you were getting better audio somehow, then the better way would be to use a universal remote that was controlling the volume on the soundbar directly.