You may need to enable 4k50/60Hz (ie 3840x2160) on your TV. Often, not all the HDMI inputs support it. You should not need those 4096x2160 (‘smpte’) modes at all.
We recommend people with 4K TV’s set their user interface (UI) to 1080p. Kodi’s UI is not optimized for 4K yet and this can put unnecessary demands on your device and can lead to a suboptimal picture quality, as well as potentially cause other issues.
The settings we recommend are as follows…
Settings>System>Display>Resolution> 1920x1080p
Settings>System>Display>Whitelist> (empty) *
Settings>Player>Videos>Adjust display refresh rate> On start/stop
Some televisions may also need, or benefit from, the following being set…
System>Display>Force 4:2:2 colour subsampling> (enable)
With the above settings your UI will be output in Full HD and your 4K content will be output in 4K. *Information regarding the whitelist can be found here. If you have any doubt, feel free to upload some logs so we can verify that your settings are indeed correct.
If you disable ‘DTS capable receiver’ in Settings->System->Audio and enable ‘Dolby Digital (AC3) capable receiver’ and ‘Enable Dolby Digital (AC3) transcoding’ that should do it. You may have to adjust the Settings level (bottom left) to see those options.